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Abstract 


The aviation system now depends on information technology more 
than ever before to ensure safety and efficiency. To address concerns 
about the efficacy of software aspects of the certification process, the 
Federal Aviation Administration (FAA) began the Streamlining Software 
Aspects of Certification (SSAC) program. The SSAC technical team was 
commissioned to gather data, analyze results, and propose 
recommendations to maximize efficiency and minimize cost and delay, 
without compromising safety. The technical team conducted two public 
workshops to identify and prioritize software approval issues, and 
conducted a survey to validate the most urgent of those issues. The 
SSAC survey, containing over two hundred questions about the FAA's 
software approval process, reached over four hundred industry software 
developers, aircraft manufacturers, and FAA designated engineering 
representatives. Three hundred people responded. This report presents 
the SSAC program rationale, survey process, preliminary findings, and 
recommendations. 

1. Introduction and Background 

The final report from the RTCA Task Force 4 on Certification states that the failure of the FAA's 
certification process to keep pace with the growth in aviation has resulted in increasing costs to the 
manufacturing industry (ref. Final). Some industry representatives have complained that the certification 
process requires an inordinate amount of time and expense, with some aspects of that process contributing 
little or no value. In particular, the FAA has received numerous complaints about software aspects of the 
certification process. To address software concerns, the FAA began the Streamlining Software Aspects of 
Certification (SSAC) program, which established a technical team to determine whether the cost and time 
associated with the software approval process can be reduced without compromising safety. 

This report presents the preliminary findings from the first industry survey on the FAA's software 
approval process, and recommendations to the FAA on ways to improve. The findings and 
recommendations presented here are preliminary because further data collection through the SSAC 
program will be needed to objectively substantiate or refute some of the results. Preliminary results are 
presented, nevertheless, to initiate interchange with the FAA and the aviation software industry in hopes 
of working together to develop an achievable plan for improvement. 

Given the preliminary nature of the findings and recommendations, this report contains only a brief 
summary of the SSAC program. Attention is focused primarily on disseminating the significant findings 
from the survey to stimulate the process of making change. A final report, to be published at the end of 
the program, will provide further detail on the SSAC program, data collection, results, and 
recommendations. 

1.1 Motivation for the SSAC Program 

Concepts for a National Airspace System (NAS) are evolving because of continuing increases in 
aviation activity and air travel worldwide. Advances in information technology provide the principal 
enabling factor for achieving a tmly unified airspace system. In a recent commentary in Aviation Week 
& Space Technology (ref. Hughes), David Hughes wrote: "Information technology is becoming a key 



part of everything the aerospace and defense industry does for a living, and as the century closes it is 
computers and software that hold the keys to the future. The industry is being transformed from 
dependence on traditional manufacturing into something that looks more like IBM and Microsoft with 
wings." On one hand, this is good because information technology makes possible new capabilities such 
as Communications/Navigation/Surveillance for Air Traffic Management (CNS/ATM), which will be the 
backbone of a unified airspace system. On the other hand, keeping up with new technology is difficult; 
doing so in a cost efficient manner is even more difficult. 

Addressing issues related to the approval of new technologies is vital to the success of the FAA's 
modernization initiatives. In particular, the FAA must grapple with two major issues, namely the safety 
aspects of the aviation system as a whole, and the cost of implementing new technologies. These two 
issues are related. According to Charles Huettner (ref. Huettner): "Currently each part of the [aviation] 
system is evaluated and certified independently, albeit remaining mindful of what the other parts are 
doing. Overall safety standards for the aviation system as a whole do not exist." In particular, there is 
little attention on certification and safety standards for CNS/ATM. Significant time and resources must 
be expended to define such safety standards. In turn, excessive cost and time burdens can negatively 
affect safety by causing delays in adopting new, safety-enhancing technologies. It is imperative that 
impact on safety be assessed, as efforts to streamline are realized. 

In short, the challenge to the FAA to keep pace with new technology, while at the same time increase 
safety and decrease cost, is great. This report focuses on the software aspects of this challenge. 

1.2 Overview of the SSAC Program 

To address concerns about software aspects of certification, the FAA commissioned an independent 
team of software engineering and safety experts to provide recommendations on ways to improve 
software aspects of certification without compromising safety. The following are members of the SSAC 
technical team: Cheryl Dorsey, independent Designated Engineering Representative from Digital Flight; 
Kelly Hayhurst, Technical Program Manager from NASA Langley Research Center; John Knight, 
professor from the Computer Science Department at the University of Virginia; Nancy Leveson, 
professor from the Aeronautics and Astronautics Department at the Massachusetts Institute of 
Technology; and, Frank McCormick, independent Designated Engineering Representative from 
Certification Services, Inc. 

The technical team is an independent group tasked to research software issues and provide 
recommendations to the FAA. The lack of direct industry participation is intentional, to reduce the 
potential for bias towards or against particular companies. The lack of direct FAA participation is also 
intentional to reduce undue influence. This does not mean, however, that the FAA is not participating 
fully. The SSAC program is sponsored jointly by FAA AIO-2 and FAA AIR- 130. Both Leanna Rierson, 
Software Program Manager FAA AIR- 130, and Ron Stroup, Software Safety and Certification Lead FAA 
AIO-200, provide guidance to the technical team and coordinate with the program sponsor, Arthur Pyster, 
Acting Deputy Assistant Administrator for Information Services and Chief Information Officer FAA 
AIO-2. 


The mission of the technical team is to gather, analyze, and synthesize objective evidence concerning 
cost and schedule drivers for software aspects of certification, and provide recommendations to the FAA 
on ways to improve without sacrificing safety. To detennine the areas of concern about software aspects 
of certification, the technical team conducted two public workshops in 1998 with industry and 
government representatives involved in software. During SSAC Workshop I, participants identified more 
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than 200 individual concerns about RTCA/DO-178B Software Considerations in Airborne Systems and 
Equipment Certification (ref. RTCA) and other aspects of the approval process. All of these were 
documented and summarized into the following 14 general issues to be considered by the technical team 
for further data collection to determine the validity of the individual perceptions expressed at Workshop I. 

1 . Inconsistencies exist among Aircraft Certification Offices (ACOs ) in interpreting and following 
policy and guidance. 

2. The effectiveness of some specific activities required by DO-178B is unclear. 

3. DO-178B inadequately addresses the effect of software on the safety of the overall system. 

4. Insufficient knowledge of software engineering and related disciplines exists within the FAA. 

5. Requirements definition is difficult independent of certification. 

6. Lack of cooperation exists between the FAA and industry. 

7. The extent to which DO-178B provides benefits beyond those that are provided by other industry 
accepted practices is unclear. 

8. Insufficient knowledge of software engineering and related disciplines exists within industry. 

9. Insufficient information is available about the certification process. 

10. Inadequacies, inconsistencies, and inefficiencies exist in the Designated Engineering 
Representative (DER) system. 

11. DO-178B inadequately provides for innovation. 

12. Problems exist within the Technical Standard Order (TSO), Type Certificate (TC), Supplemental 
Type Certificate (STC), Amended Type Certificate (ATC), and Parts Manufacturing Approval 
(PMA) processes. 

13. Lack of cooperation among companies increases costs. 

14. Working with certification authorities outside of the United States (e.g., the European Joint 
Aviation Authorities) is difficult. 

For brevity, this report does not provide the detailed individual concerns behind these general issues. 
For a complete listing of the concerns expressed at Workshop I and the summarization strategy, see 
Streamlining Software Aspects of Certification: Technical Team Report on the First Industry Workshop 
(ref. Hayhurst). 

Given the broad scope of each of the 14 general issues, extensive data collection to substantiate all of 
them would burden both the collector and respondent. Consequently, SSAC Workshop II was held to 
determine the issues industry considered to be the most important to study. Each Workshop II participant 
was asked to identify the five most important issues for the technical team to investigate. The results 
were collected and the 14 issues were ranked from most to least important. 

Based on this ranking and recommendations from the FAA, the technical team pursued further data 
collection in the following areas. 

• Interactions with ACOs and other approving authorities 

- Issue 1. Inconsistencies exist among ACOs in interpreting and following policy and guidance. 

- Issue 6. Lack of cooperation exists between the FAA and industry. 

- Issue 9. Insufficient information is available about the certification process. 
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• Specific activities required by DO-178B, including independence, structural coverage, traceability, 

documentation, quality assurance, and tool qualification 

- Issue 2. The effectiveness of some specific activities required by DO-178B is unclear. 

• Safety 

- Issue 3. DO-178B inadequately addresses the effect of software on the safety of the overall 
system. 

• The DER system 

- Issue 10. Inadequacies, inconsistencies, and inefficiencies exist in the DER system. 

There were no mechanisms in place to assure that those attending the SSAC workshops constituted a 
fair representation of the aviation software community as a whole. Consequently, the workshop 
information alone does not provide a sufficient basis for developing recommendations for potential 
changes to the FAA's software approval process. Instead, the data gathered at the workshops served to 
hone the data collection process to focus on the most urgent problem areas. 

1.3 SSAC Survey 

The next step in data collection was to assure that the issues identified as most important to the 
workshop participants were indeed significant issues for the general population that develops airborne or 
ground-based systems containing software (DO-178B compliant software in particular). Collecting data 
from the entire population was clearly infeasible. Hence, data from a subset of the general population 
was needed. To make valid statistical inferences from data collected from a subset of the population, the 
data must come from individuals that together forni a reasonable representation of the population of 
airborne and ground-based software developers that must comply with DO-178B. 

Surveys provide a relatively inexpensive, yet effective means for getting information from a large, 
geographically dispersed population. Although some doubt the reliability of data derived from a 
relatively small number of respondents, sample survey methods have gained wide acceptance for their 
accuracy in producing standardized data amenable to quantification and statistical analysis. Surveys are 
also cost effective because they impose only limited time constraints on respondents and do not require 
time and travel resources to attend meetings. Additionally, surveys can ensure anonymity and 
confidentiality (which is important when commenting on a regulatory authority such as the FAA). 

Based on these considerations, the technical team conducted an industry-wide survey. The survey was 
designed to detennine overall satisfaction in the aviation industry with the FAA’s software approval 
process, and detennine the extent and significance of the problem areas identified at the workshops. The 
technical team sought the expertise of the Center for Survey Research (CSR) at the University of Virginia 
to assist in the survey process. The four elements in this process were identification of the sample 
population, development of the survey instrument, administration of the survey, and data analysis. 

1.3.1 Identification of Sample Population 

To make generalizations and statistical inferences about the software aviation community, the sample 
population must be a reasonable representation of the community. The following attributes were 
detennined to be important for ensuring that the SSAC survey population constituted a reasonable 
representation: 
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• different levels of experience with DO-178B; 

• different roles in the development and approval process for software; 

• experience with a variety of airborne systems products installed on a variety of different aircraft 
and engine types; 

• experience with a variety of ground-based systems products; and, 

• experience with projects of various size and criticality. 

Statisticians prefer to choose a sample population by random selection methods; however, there are no 
public listings of individuals involved in the development and approval of software in compliance with 
DO-178B. Thus, random methods could not be applied in this case. Consequently, the sample 
population was identified on a company basis to attempt to ensure a representative sample and to ensure 
that company management was aware of the effort. Because much of the survey content concerns FAA- 
specific issues, participation was limited to companies within the United States. 

The technical team distributed survey infonnation to companies with experience developing DO-178B 
compliant software. These companies were identified through attendance lists from SSAC workshops, 
RTCA membership lists, and lists provided by FAA offices. 

Because the survey issues were specific to DO-178B, the minimum criteria for survey participation 
was experience with DO-178B in at least one of the following roles: 

• software engineer with direct development responsibility in software design, testing, or quality 
assurance; 

• software engineering project manager responsible for directing software professionals; 

• system project manager responsible for the overall cost and schedule including software; or, 

• certification liaison, including company or consultant DER, responsible for coordination with the 
ACO or other approving authority. 

Each company interested in participating in the survey identified a point-of-contact to register 
qualified individuals within the company to receive the survey and to coordinate distribution of the 
survey. Because company size and product lines vary throughout the industry, no limits were placed on 
the number of individuals in each company that could register. A total of 416 individuals (representing 
more than 70 companies) registered for the survey through this process. 

1.3.2 Survey Instrument and Implementation 

The technical team defined the questions to be included in the survey instrument. The final survey 
questionnaire is given in Appendix A. For the most part, the questions came directly from the issues 
raised at Workshops I and II. The survey questions divided into the 15 topics shown in Table 1 . 

To improve the quality of the survey, a number of steps were taken prior to producing the 
questionnaire. The survey questions were reviewed by CSR to help remove bias in the way questions 
were stated and ensure that the response options were independent and complete. FAA representatives 
reviewed the questions to ensure that their concerns were addressed. Finally, the technical team 


Although the approval processes for ground-based and airborne systems differ substantially, software aspects of 
approval, in particular development in compliance with DO-178B. are similar. 
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conducted two pretests with small subsets of the population. The pretest participants provided feedback 
on the questionnaire's clarity and comprehensiveness. 


Table 1. Organization and Content of the SSAC Survey Questionnaire 


Survey Section 

Topic 

A & B 

Respondent Background and Experience 

C 

FAA Policy, Guidance, and Audits 

D 

Aircraft Certification Offices 

GD 

Approving Authorities for Ground-based Systems 

E 

Independence 

F 

Modified Condition Decision Coverage 

G 

Traceability 

H 

Quality Assurance 

I 

Documentation 

J 

Tool Qualification 

K 

Safety 

L 

Designated Engineering Representatives 

GL 

Using Designated Engineering Representatives for Ground-based Systems 

M 

Availability of Information about the Certification Process 

P 

Appropriateness of DO-178B for Ground-based Systems 


Changes identified during reviews and pretests were made before the final survey questionnaire was 
produced. The questionnaire was then coded on a floppy disk using a Computer Assisted Self- 
administered Interviewing (CASI) format. CSR administered the survey, including survey distribution 
and follow-up reminders, over a 10-week period from mid-December 1998 through mid-February 1999. 
To help allay fears about commenting on an approving authority, CSR collected all of the responses and 
filtered identifiers, such as name and company affiliation, before delivering the survey responses to the 
technical team. During the collection period, CSR received 300 questionnaires, for a response rate of 
approximately 72%. Of those 300 questionnaires, 292 were completed surveys suitable for analysis. 

2. Survey Results 

The presentation of the survey results follows the outline of the survey. Except for the respondent 
profile, each section begins with an overview of the background and objectives for that section, followed 
by the significant findings and technical recommendations. While many of the specific recommendations 
are intended for the FAA, some recommendations are intended for the RTCA Special Committee 190 on 
Software Application Guidelines for RTCA DO-178B/ED-12b or for the industry in general. 

Relevant survey questions are identified in brackets throughout the discussion. The reader may 
consult Appendix A to find the text of each question. Specific survey responses are italicized. Unless 
otherwise noted, all of the survey respondents answered the questions in each survey section. 

2.1 Sections A and B: Profile of Survey Respondents 

Sections A and B of the survey questionnaire yielded a profile of each survey respondent, from which 
could be established whether the sample population was representative of the aviation software 
community as a whole. As mentioned previously, the survey specifically targeted engineers and 
managers, airborne and ground-based systems developers, and manufacturers and product suppliers, all 
with a range of experience with DO-178B. 
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Because the approval process is different in some aspects for airborne and ground-based systems, 
certain sections of the survey targeted issues related only to airborne systems and others targeted issues 
for ground-based systems. Accordingly, the sample population was divided into airborne and ground- 
based respondents. Of the respondents representing the 292 completed surveys used in this preliminary 
analysis, 206 were considered airborne respondents and 86 were considered ground-based respondents 
[A4, A5], For the purposes of this survey, respondents with both airborne and ground-based experience 
within the past 5 years were considered as ground-based respondents. A majority of ground-based 
respondents also had experience with airborne systems within the past 5 years. 

2.1.1 Respondent Experience 

In general, the survey indicated that the respondents had considerable experience in aviation software. 
Over 65% of the respondents cited 10 or more years of experience in aviation software [A1 ]. Over 58% 
of the respondents had been with their current employer at least 10 years [A2], Only 15 respondents had 
less than 2 years of experience in aviation software. 

Many of the respondents had multiple roles in the development and approval of software. Just over 
75% of the respondents cited extensive software engineering experience within the past five years [A3a], 
Most of the respondents also had experience with some level of software management responsibility as a 
software engineering lead or project manager. Almost 72% of the respondents had at least some project 
management experience [A3c]. Of the 292 respondents, 194 had at least some certification liaison 
experience [A3d], Of those claiming certification liaison experience, more than half were not DERs, 56 
were DERs, and 18 were candidate DERs [A3f], Over 93% of the DERs were company DERs, and most 
(about 75%) could approve Level A data [A3g]. 

Although most of the respondents had substantial experience in aviation software, most of the 
respondents did not have experience on a large number of DO-178B projects. This was expected given 
the relatively short time DO-178B has been in use. About 71% had completed five or fewer projects. 
Almost 20% had not completed a DO-178B project yet [B7], However, most of the respondents did have 
experience developing software under standards other than DO-178B. Approximately 73% had personal 
experience with DO-178A [B8a], about 45% had experience with military standards [B8d], and 27% had 
experience with other standards [B8e], Experience within software development teams was similar to the 
personal experience. 

About 84% of the respondents had at least some experience with Level A or B software [BIO], A 
significant number of the respondents (about 72%) dealt with Level A or B at least regularly. The size of 
the largest current software project ranged from less than 1000 non-commented source lines to over 
350,000 non-commented source lines [B 12], The survey data did not provide a clear picture of the 
amount of software developed for each software level. 

2.1.2 Company Information 

As expected based on the general population, most respondents (about 75%) worked for airborne 
equipment suppliers [A6], Of the remaining 25%, about 9% worked for aircraft manufacturers, 4% 
worked for ground equipment suppliers, 4% worked for suppliers of both airborne and ground equipment, 
and about 5% worked for engine manufacturers. Approximately 54% of the respondents indicated they 
knew the Capability Maturity Model (CMM) rating for their company [B 16], Of those, more than 90% 
indicated their company was rated at Level 2 or higher. About 56% were at Level 2 and 18% at Level 3. 
About 68% of those ratings were determined by external assessment [B17], 
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The respondents had experience with a wide range of products for both airborne [B3] and ground- 
based [B6] systems, as shown in Tables 2 and 3. 


Table 2. Respondent Experience with Airborne Systems Products 


Airborne product 

Number of respondents with experience 

Autopilot 

45 

Flight management computer 

50 

Engine control 

46 

Passenger entertainment 

10 

Displays 

74 

Navigation equipment 

63 

Flight controls 

43 

Anti-skid braking 

12 

Fuel systems 

28 

Electrical/power systems 

24 

Communication equipment 

48 

Audio systems 

16 

Traffic collision avoidance systems 

10 

Data acquisition 

5 

Weather-related systems 

8 

Monitoring & maintenance 

11 

Other 

25 


Table 3, Respondent Experience with Ground-based Systems Products 


Ground-based systems product 

Number of respondents with experience 

Communications 

27 

Navigation 

41 

Surveillance 

12 

Automation 

16 

Air traffic management 

12 


4 

Maintenance and test 

6 

Information/data management 

4 

Other 

6 


The airborne products were also associated with a variety of different aircraft and engine types [B4], as 
shown in Table 4. 


Table 4. Aircraft or Engine Type Associated with the Airborne Systems Product Lines 


Aircraft or engine type 

Number of respondents with experience 

Transport aircraft 

121 

Regional aircraft 

100 

Business aircraft (Part 91) 

119 

General aviation aircraft 

67 

Rotorcraft 

52 

Large turbine engine 

41 

Small turbine engine 

54 

Piston engine 

15 

Military aircraft 

3 

Other 

3 
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The size of software development teams and the number and size of the software projects varied 
significantly over the sample population, as expected given the diversity throughout the aviation software 
industry. Most (68%) of the respondents worked on relatively small software development teams with 15 
or fewer members [B2], Less than 9% of the respondents worked on large teams with more than 50 
members. About 46% of the respondents worked with companies that had five or fewer DO-178B 
projects approved per year [B 13]. More than 50% of the respondents received TSO authorizations on at 
least 76% of their projects [B5], 

The final measure of whether the sample population for airborne respondents was representative of the 
general population was the geographic distribution of the respondents. All of the Certification 
Directorates, and all but two of the ACOs in the United States (in Denver and Anchorage), were 
represented by the survey respondents [B14], Most of the airborne respondents worked with the Small 
Airplane or Transport Airplane Directorates. In about 85% of the cases, the primary ACO was the one 
that covers the geographic region [B14A], In addition to their primary ACO, respondents indicated that 
they interacted with other ACOs; frequently mentioned ACOs were Seattle, Wichita, Los Angeles, and 
Atlanta. Table 5 shows the primary and secondary ACOs specified by the airborne respondents. 


Table 5. Airborne Respondents' Primary and Secondary Aircraft Certification Directorates and Offices 


Certification Directorate 
Certification Office 

maun 

Number of respondents 
indicating secondary ACO 

Transport Airplane Certification Directorate 

77 

98 

Denver Aircraft Certification Office (ANM-100D) 

0 

1 

Los Angeles Aircraft Certification Office (ANM-100L) 

48 

42 

Seattle Aircraft Certification Office (ANM-100S) 

29 

55 

Small Airplane Certification Directorate 

83 

99 

Anchorage Aircraft Certification Office (ACE 1 15N) 

0 

2 

Atlanta Aircraft Certification Office (ACE 115 A) 

8 

37 

Chicago Aircraft Certification Office (ACE-1 15C) 

25 

13 

Wichita Aircraft Certification Office (ACE-1 15W) 

50 

47 

Rotor era ft Certification Directorate 

4 

17 

Fort Worth Airplane Certification Office (ASW-150) 

1 

5 

Fort Worth Rotorcraft Certification Office (ASW-170) 

1 

10 

Fort Worth Special Certification Office (ASW-190) 

2 

2 

Engine and Propeller Certification Directorate 

32 

28 

Boston Aircraft Certification Office (ANE-150) 

17 

7 

Engine Certification Office (ANE-140) 

7 

8 

New York Aircraft Certification Office (ANE-170) 

8 

13 

Brussels Aircraft Certification Division (AEU-100) 

0 

8 


In developing this survey, the team expected that for most of the ground-based systems developed 
using DO-178B, one of three primary approving authorities would be involved: (1) the Product Team for 
the Global Positioning System (FAA AND-730); (2) the Product Team for Navigation and Landing (FAA 
AND-740); or (3) the GPS, Navigation and Landing Branch (FAA AOS-240). Although many of the 
ground-based respondents indicated that they worked with these three approving authorities, they also 
cited experience working with several ACOs including Los Angeles, Seattle, Chicago, Engine 
Certification, Wichita, Boston, and New York. Table 6 shows the distribution of approving authorities 
for ground-based systems. 
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Table 6. Approving Authorities Indicated by the Ground-based Respondents 


Approving authority for ground-based equipment 

Number of respondents with 
experience 

Product Team for the Global Positioning System ( AND-730) 

17 

Product Team for Navigation and Landing (AND-740) 

17 

GPS, Navigation and Landing Branch (AOS-240) 

28 

Los Angeles Aircraft Certification Office (ANM-100L) 

5 

Seattle Aircraft Certification Office (ANM-IOOS) 

3 

Engine Certification Office (ANE-140) 

3 

Chicago Aircraft Certification Office (ACE-1 15C) 

2 

Wichita Aircraft Certification Office (ACE-1 15W) 

2 

Boston Aircraft Certification Office (ANE-150) 

1 

New York Aircraft Certification Office (ANE-170) 

1 

Fort Worth Special Certification Office (ASW-190) 

1 

Other 

3 


Overall, the respondent profile showed that the survey reached a wide cross section of the aviation 
software community. Although there are no definitive statistics on various attributes of the population for 
comparison, the responses to the profile questions provided sufficient evidence that the sample population 
was a fair representation of the general population. Thus, the team believes that drawing generalizations 
from the survey findings is reasonable. 
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2.2 Section C: FAA Policy, Guidance, and Audits 

A major area of concern raised by attendees of the two SSAC workshops was software policy and 
guidance. This area has the potential for affecting costs significantly. Workshop attendees claimed that 
policies are unclear and that FAA guidance in several technical areas is incomplete. This section of the 
survey addressed the guidance received by the industry on technical and process issues concerned with 
certification and with audit timing. Both airborne and ground-based respondents were asked to answer 
the questions in this section. 

The section was divided into four major parts: 

(1) overall satisfaction with the certification process and with written software policy and guidance; 

(2) specific issues pertaining to different aspects of DO-178B guidance; 

(3) specific issues pertaining to "additional considerations" not explicitly related to DO-178B and 
which require FAA policy and guidance; and, 

(4) audit frequency and duration. 

The following were the objectives for this section. 

• Determine an overall level of satisfaction with software aspects of certification 

• Determine if regulatory software reviews (on-site audits) impose undue burden without benefit 

2.2.1 Significant Findings 

The section began with a general question about overall satisfaction with the software approval 
process [Cl], The survey responses for this question were encouraging: 70.4% of the respondents 
selected a positive or neutral response (very satisfied, somewhat satisfied, or neither satisfied nor 
dissatisfied ). A purely positive interpretation of the responses to this question is, however, inconsistent 
with the survey responses to the detailed questions in this section. Also, the fact that nearly 30% of the 
respondents were either somewhat dissatisfied (23.6%) or very dissatisfied (6%) — indicates room for 
improvement. 

A detailed analysis of the responses to question Cl yielded additional insight but little change in the 
conclusion. Analysis of the responses was performed to detemrine overall satisfaction of various groups 
of respondents with the FAA's software approval process as shown in Table 7. 


T able 7. Satisfaction of Various Subgroups with the FAA Process for Software Aspects of Certification 


Subgroups of the sample population 

Percentage of respondents either 
somewhat or very dissatisfied 

Experience of 1-5 certifications using DO-178B 

26% 

Experience of 6 or more certifications using DO-178B 

31% 

Regular experience with Level A or B software 

28% 

Most experience with Level A or B software 

28% 

Extensive project management experience 

36% 

Some project management experience 

24% 

Extensive software engineering experience 

27% 

Some software engineering experience 

35% 
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This table shows that dissatisfaction with the FAA process for software aspects of certification is not 
related to lack of experience or training. Thus, the overall dissatisfaction level cannot be legitimately 
dismissed for this reason. 

Question C2A asked how satisfied respondents were in general (overall) with existing, written FAA 
software policy and guidance. For the survey, "written policy and guidance" was defined as including the 
Federal Aviation Regulations (FARs), DO-178B, FAA Notices, FAA Orders, Advisory Circulars, and 
FAA policy memos. The majority of respondents were either somewhat satisfied (39.9%) or very 
satisfied (2.4%). A total of 27.3% were neither satisfied nor dissatisfied. But very large fractions of the 
respondents were somewhat dissatisfied (23.8%) or very dissatisfied (6.6%), justifying the conclusion that 
the quality of written software guidance can be improved. 

As with question Cl, a detailed analysis of the responses to question C2A yielded insight but little 
change in the conclusion. Analysis of the responses was performed to determine overall satisfaction of 
various groups of respondents with software policy and guidance as shown in Table 8. 


Table 8. Satisfaction of Various Subgroups with FAA Software Policy and Guidance 


Subgroups of the sample population 

Percentage of respondents either 
somewhat or very dissatisfied 

Experience of 1-5 certifications using DO-178B 

31% 

Experience of 6-20 certifications using DO-178B 

23% 

Regular experience with Level A or B software 

29% 

Most experience with Level A or B software 

33% 

Extensive project management experience 

40% 

Some project management experience 

28% 

Extensive software engineering experience 

25% 

Some software engineering experience 

47% 


Again, this data confirmed the conclusion that the level of dissatisfaction was fairly high, and that this 
was not a result of a lack of software engineering or DO-178B experience or training. 

Sixteen questions asked about the adequacy of documented software policy in specific technical areas 
[C2a-q], A final question asked about areas not covered by the previous 16. The technical areas covered 
were: 


• planning [C2a]; • Commercial-Off-The-Shelf (COTS) software [C2i]; 

• requirements [C2b]; • legacy software and software reuse [C2j]; 

• design [C2c]; • user-modifiable software [C21]; 

• code [C2d]; • partitioning [C2m], 

• verification [C2e]; • service history [C2n]; 

• configuration management [C2f]; • tool qualification [C2o]; 

• quality assurance [C2g]; • classification of changes [C2p]; and, 

• certification liaison [C2h]; • alternate means of compliance [C2q], 


The responses to these questions were remarkably consistent and mostly negative. For questions C2a 
through C2h about the software life cycle processes in DO-178B, the dominant response was barely 
sufficient. When combining barely sufficient with the next category, insufficient, the percentage in these 
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two categories was at least 85% in all cases. For questions C2i through C2q about additional 
considerations, the dominant response was insufficient. In almost all cases, the percentage of respondents 
claiming insufficient software policy in this category was more than 50%. 

Thus, the survey results showed that the present written software guidance for some important 
technical areas is inadequate. 

Question C2r asked for comments about technical areas not addressed by other questions in this 
section. A total of 50 comments were received. The topics covered many aspects of the certification 
process, including the following: 

(1) inconsistent interpretation of documentation about the certification process; and, 

(2) lack of guidance, or unawareness on the part of the respondents that guidance is available, in a 
number of areas (for example, reuse, structural coverage, and safety). 

Survey questions C3 (for airborne respondents) and GC3 (for ground-based respondents) asked 
whether the respondent had participated within the last five years in a software audit; subsequent 
questions asked the respondent's view about timing and depth of these audits. Roughly 50% of the 
respondents had participated in at least one audit. 

Question C5 asked the respondent for an opinion about the appropriateness of the number of audits. 
Almost 62% replied that the number of audits was about right. In-depth analysis of the responses to this 
question revealed that the number of respondents who chose about right varied considerably with factors 
such as software engineering experience, but in no case did it drop below 50%. Question C7 asked about 
the duration of regulatory audits. Seventy-nine percent of the responses were about right. 

Question CIO was about the depth of software audits. Roughly equal numbers of respondents thought 
the depth of regulatory audits was about right (42.8%) or vary significantly (42.8%). The high percentage 
of responses that cited significant variation in the depth of regulatory audits indicates this should be 
further understood and addressed. 

2.2.2 Summary and Recommendations 

Based on the survey responses, the written software policy and guidance provided for the certification 
process appears to be inadequate in both availability and quality. This is a serious situation that appears 
to lead to major project delays and unnecessary expenditures by aerospace companies. Revising the 
guidance documents and improving the process by which they are developed would help. With respect to 
software audits, the timing appeared to be satisfactory for the most part. The most significant issue was 
the variability of the depth of regulatory audits. 

Recommendation Cl: The FAA and the industry, in conjunction with RTCA, should determine the 
appropriate means for providing infonnation for all software life cycle processes. For example, the FAA 
and the industry should determine what is needed to supplement and clarify DO-178B in the areas of 
requirements, design, code, verification, planning, certification liaison, quality assurance, and 
configuration management. 

Recommendation C2: The FAA should lead the development of policy and guidance for COTS 
software, legacy software and software reuse, user-modifiable software, partitioning, tool qualification, 
service history, major and minor software changes, alternate means of compliance, and emerging 
technologies as they arise. 
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Recommendation C3: The FA A should develop an approach to ensure consistency in the depth of 
regulatory audits at the different software levels. 
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2.3 Section D: Aircraft Certification Offices 


This section of the survey was designed to assess issues of inconsistencies and inadequacies specific to 
approving authorities. Because airborne and ground-based respondents typically interact with different 
approving authorities, questions for this topic were partitioned into two parts: Section D was devoted to 
airborne respondents' experience with ACOs, and Section GD was devoted to ground-based respondents' 
experience with their primary approving authorities. 

Certification is not an easy task, requiring considerable cooperation and communication between 
regulator and applicant. At Workshop I, participants expressed concerns regarding interactions with 
ACOs and other approving authorities. Their concerns were generally in two areas, communication 
(ranked as the sixth most important issue) and consistency (ranked as the most important issue). 

The Task Force 4 report cites "poor quality or slow communication between authorities' offices and 
particularly across organizational boundaries within an authority" (ref. Final, page 84). SSAC workshop 
participants also thought that ACOs could improve the timeliness of feedback and coordination of 
certification issues with national resources. Honoring agreements was also a primary concern. This 
concern is echoed in the Task Force 4 finding that decision reversals may occur within the FAA without 
adequate involvement of technical personnel (ref. Final, page 84). 

On the consistency issue, concerns were expressed about inconsistencies that exist between ACOs 
with regard to software policy, guidance, and the interpretation of DO-178B. These concerns prompted 
allegations that applicants were moving from one ACO to another to shop for favorable circumstances. In 
addition, workshop attendees thought ACO engineers within the same ACO treat some companies 
differently. Regardless of the source of the inconsistency, workshop participants were vocal in their call 
for a level playing field. Only those airborne respondents indicating they had experience interacting with 
their primary ACO were prompted to answer the questions in this section. 

The following were the objectives for this section. 

• Detennine level of satisfaction with services provided by ACOs 

• Determine if there have been instances of inconsistencies among or within ACOs and determine if 
there have been more than a few isolated occurrences 

• If there are frequent or substantial accounts of inconsistencies, detennine if they impact cost or 
schedule 

• Detennine what types of inconsistencies exist 
2.3.1 Significant Findings 

Of the 206 airborne respondents, 77 (38%) believed they had enough interaction with their primary 
ACO to rate its perfonnance [D 1 ]. These 77 are the "section D respondents". Section D respondents had 
a split opinion about the overall satisfaction with their primary ACO. Roughly, 54% were either 
somewhat satisfied or very satisfied with their primary ACO, while 26% were somewhat or very 
dissatisfied [D2], The remaining questions in this section addressed the communication and consistency 
issues in detail. 


15 



2.3.1.1 Communication 


Survey respondents were queried about specific communication and coordination activities related to 
software. Most respondents indicated broad satisfaction (defined as 60% or more reporting good, very 
good, or excellent experiences with the FAA) with their primary ACO's ability to return phone calls 
[D3a], respond to written communication [D3b], address certification issues [D3c], and coordinate 
meetings and reviews [D3e], Areas of most concern were establishing and honoring dates for reviews 
and approvals [D3d], with 20% rating their primary ACO as poor, coordination with technical resources 
[D3g], with a 17% poor rating; and, response to submitted plans [D3h], with a 22% poor rating. 

Honoring verbal and written agreements was of particular concern to workshop participants. Survey 
respondents indicated that ACOs reneged on both verbal and written agreements. Exactly 50% indicated 
that verbal agreements had not been honored [D25], and 20% indicated that written agreements had not 
been honored [D26], The explanation cited most frequently was FAA personnel changed [D27], This in 
itself suggests that ACO engineers looking at the same data often do not make the same decisions. 

2.3.1.2 Consistency 

The survey results indicated that inconsistencies exist within and among ACOs. Of the section D 
respondents, 52 of them (about 67%) worked with more than one ACO [D7]. Of these 52 respondents, 39 
of them (76.5%) said they received inconsistent software guidance, interpretations, or procedural 
requirements from different ACOs within the past three years [D9], with 61% incurring major additional 
costs as a result [D13a], Of the 39 respondents experiencing inconsistencies, less then 13% claimed they 
happened rarely [D12], and almost 53% indicated that the inconsistency was fundamentally different 
[D13]. According to the survey, the differences with greatest consequence were about issue papers and 
interpretation of DO-178B [DlOb], The survey responses indicated that inconsistencies and 
communication issues with ACOs did not result in applicants shopping for more favorable certification 
circumstances. The major reason cited for using multiple certification offices was customer’ s geographic 
region [D8J. 

Of the 77 section D respondents, 36% received inconsistent software guidance within the same ACO 
[D20], Although the number of respondents indicating such experience was relatively small (26), the 
survey results showed that when these inconsistencies occurred, they were not isolated incidents. Of the 
26 respondents that experienced inconsistency from within the same ACO, most of them experienced 
such at least occasionally or frequently [D22] and incurred major additional costs as a result [D23], As in 
the case of inconsistency among different ACOs, inconsistent interpretation of DO-178B was the single 
most frequently cited problem [D24], 

Giving respondents the opportunity to comment about their primary ACO provided insights. Many 
claimed that their ACO is understaffed, the workload is too high, and the ACO engineers have 
insufficient background in software [D4], This supports the Task Force 4 finding that FAA employees 
are "currently being spread too thin" (ref. Final, page 84). With respect to consistency, one respondent 
suggested that "inconsistencies lead to reduced levels of safety, with the least safe systems and equipment 
being the most competitive due to reduced regulatory overhead." Another stated "the main problem is a 
lack of a level playing field between ACOs." Other comments stated that differences exist among the 
National Resource Specialist (NRS), ACOs, Certification Directorate, and FAA Headquarters. 
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2.3.2 Summary and Recommendations 

The survey results revealed problems in communication between ACOs and applicants and 
inconsistency in interpretation of software policy and guidance, especially inconsistent interpretation of 
DO-178B. Although the number of respondents experiencing inconsistencies among and within ACOs 
was not large, inconsistencies did occur and imposed major additional costs. 

Recommendation Dl: The FAA should identify the minimum software capability needed within the 
agency to efficiently and effectively communicate and coordinate with applicants, and implement 
appropriate hiring, policy, and training as needed. 

Recommendation D2: The FAA should examine the circumstances leading to nullification of verbal and 
written agreements. 

Recommendation D3: The FAA and applicants should document agreements up front. The FAA should 
develop and implement processes and supporting policy for managing agreements. 

Recommendation D4: The FAA should investigate the root cause(s) of inconsistencies in software 
guidance, interpretation, and procedural requirements among and within ACOs. 
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2.4 Section GD: Approving Authorities for Ground-based Systems 

This survey section examined industry’s perceptions of the FAA offices with responsibility for 
approving ground-based equipment containing software. In particular, AND-730 for the Local Area 
Augmentation System and the Wide Area Augmentation System, AND-740 for Instrument Landing 
Systems and the Transponder Landing System, and AOS-240 for Special Category I ground stations 
formed most of this group. Any ACO involvement with this segment of the industry was excluded from 
the discussion in this section. Only ground-based respondents were prompted to answer the questions in 
this section. 

In accord with Section D, the following were the objectives for this section. 

• Determine level of satisfaction with services provided by approving authorities for ground-based 
systems 

• Detennine if there are inconsistencies among approving authorities and, if so, determine if these 
inconsistencies are common 

• If there are frequent or substantial accounts of inconsistencies, determine if they affect cost or 
schedule 

• Determine what types of inconsistencies exist 

2.4.1 Significant Findings 

Only a small subset of the sample population answered the questions in this section. Although 86 
survey respondents identified themselves as having ground-based experience, barely a quarter of those 
(22 respondents) reported direct contact with the three FAA organizations of primary interest: AND-730, 
AND-740, and AOS-240 [GDI], These 22 are referred to as "section GD respondents". The small 
number of the section GD respondents precludes extrapolation to a larger population; but many of the 
trends shown in survey section D were also reflected here, which lends credibility to the results. 

There was a noticeable split in the overall levels of satisfaction with the approving authorities for 
ground-based equipment in this context: fairly heavy at each end of the satisfaction spectrum, with little 
neutrality [GD3], Almost 59% of the section GD respondents were either very or somewhat satisfied with 
their approving authority, while almost 30% said they were very or somewhat dissatisfied. Less than 12% 
said they were neither satisfied nor dissatisfied. The bulge seen at the dissatisfied end of the spectrum 
has several possible explanations — mismatched expectations between applicants and regulators, 
inexperience of either or both, and so on — and merits further examination. 

2.4.1.1 Communication 

Question GD4 was composed of several questions aimed at the overall satisfaction with the approval 
authority, especially with respect to communication and coordination. Within this block of responses, 
two patterns emerged. The first was one of broad satisfaction (60% or more of the section GD 
respondents reporting good, very good, or excellent experiences with the FAA). This pattern was 
associated with FAA responsiveness via telephone [GD4a], responsiveness to written communications 
[GD4b], perfonnance in addressing certification issues [GD4c], performance in establishing and honoring 
calendar commitments [GD4d], and perfonnance in coordinating meetings and reviews [GD4e], The 
remaining questions of this section — coordination with technical resources outside the approving 
authority’s immediate group [GD4g] and handling of submitted plans [GD4h] — revealed marginally 
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greater dissatisfaction, with fair and poor responses ranging from 35-50%. The same was noted in the 
airborne section. 

Regardless of the number of incidents, nullification of previously agreed positions can be disruptive 
and expensive, and is taken seriously by both the FAA and applicants. Although the absolute numbers 
were not large, ground-based respondents indicated incidents where verbal and written agreements had 
not been honored. Ten of the section GD respondents indicated that verbal agreements with ground- 
based authorities had been nullified [GD21], and five said that written agreements had been nullified 
[GD22], The survey responses revealed a pattern of similarities between air and ground in the reasons for 
nullification of agreements. The three leading reasons given by respondents for these annulments were 
FAA rules changed, FAA personnel changed, and the original FAA parties to the agreement rescinded it 
[GD 23], 

When survey participants were asked to provide commentary on their experience working with AND- 
730, AND-740 and AOS-240 [GD5], the responses also fell broadly into opposite ends of the spectrum. 
Some respondents indicated dismay with perceptions of unclear expectations by the FAA, lack of 
appropriate policy instruments, a "checklist" approach to approvals, and excessive workload imposed on 
FAA personnel. Conversely, a few respondents were equally clear in their praise for FAA 
professionalism and helpfulness. Results were thus mixed, but these responses were somewhat less 
positive than one might expect from the levels of satisfaction seen earlier. 

2.4. 1.2 Consistency 

To find respondents most qualified to judge inconsistencies among different FAA organizations, 
Question GD6 asked if the respondent had worked with both AOS-240 and either of the AND groups. 
Only three respondents answered yes to this question. Of those three, each worked with both 
organizations out of necessity [GD7]. Possibly of greater significance, despite the tiny sample size, was 
that all three reported inconsistencies in guidance, interpretation or procedural requirements from the 
FAA organizations involved [GD8], 

2.4.2 Summary and Recommendations 

For the ground-based respondents, problems in communication were more evident than problems with 
consistency, likely due to the small subset responding to this section. The following recommendations 
address the communication issue. 

Recommendation GDI: The FAA should examine the circumstances leading to nullification of verbal 
and written agreements. 

Recommendation GD2: The FAA and suppliers should document agreements up front. The FAA 
should develop and implement processes and supporting policy for managing agreements. 
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2.5 Section E: Independence 


DO-178B requires that some activities be performed "with independence"; that is, that there is some 
separation of responsibilities which ensures that specific objectives in Annex A of DO-178B are 
accomplished. According to DO-178B, independence for a verification activity means that the 
verification of an artifact is done by a different person than the one who developed the artifact itself. At 
Workshop I, some participants expressed concerns about the contribution of the independence 
requirements to safety and product quality. Some participants were also unclear about the meaning of the 
term "independence". Both airborne and ground-based respondents were prompted to answer the 
questions in this section. 

The following were the objectives for this section. 

• Detemiine level of understanding of the independence requirements 

• Detennine if the requirements for independence provide a tangible benefit 

2.5.1 Significant Findings 

To detemiine whether the respondents understood the definition of independence correctly, question 
El asked if independence requires verification by a different person or by a different organization. About 
86% of those who answered the question knew that independence only required a different person, not a 
different organization. There was no statistical correlation between understanding the definition of 
independence and either level of experience with DO-178B or software engineering experience. 

Respondents were asked their opinions about overall satisfaction with the independence requirements 
[E2], In general, satisfaction was high, with 63% reporting to be either somewhat or very satisfied with 
the independence requirements and 28% neither satisfied nor dissatisfied. Only 8.5% reported that they 
were somewhat dissatisfied with the independence requirements, and only one person (out of 272 people 
who responded to this question) was very dissatisfied. 

Of the respondents who answered the question about the value of independence [E8], 82% thought 
that it was extremely valuable (26%) or somewhat valuable (56%). Only 13% thought that independence 
was of marginal value, and only 1.8% (5 out of 280) thought it was of no value. There was no correlation 
between level of experience with DO-178B and belief about the value of the independence requirements. 
The respondents with the most involvement with Level A or Level B software were most likely to answer 
that the independence requirements were extremely or somewhat valuable. 

Respondents were also asked how much time and cost their company would devote to independence if 
it were not required by DO-178B [E7], More than one half of the respondents (58.5%) said that they 
would devote the same or almost as much as now, while 27% said that they would devote substantially 
less than now, and 3% said they would devote no effort to independence if it were not required. 

A requirement may be considered valuable, but it also may be very costly. Questions E5 and E6 were 
asked to detennine the cost differential between meeting a DO-178B requirement and meeting that 
requirement with independence. Of those able to estimate the costs of independence, 43.5% indicated 
that the software development costs attributed to independence were negligible or small, 47% that the 
costs were substantial, and only 1% that the costs were nearly prohibitive. Similarly, 44% stated that the 
software development time attributed to independence was negligible or small, 48% thought it was 
substantial , and only one person (out of 270 respondents to this question) thought it was nearly 
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prohibitive. The individual who reported nearly prohibitive costs and time reported only rare 
involvement with Level A or Level B software. The survey results showed that the costs attributed to 
independence decreased as experience with DO-178B increased. 

In trying to ascertain which independence requirements might be most bothersome, respondents were 
asked to indicate which requirements they did not like [E3]. These comments further demonstrated some 
respondents' confusion about the meaning of independence. 

2.5.2 Summary and Recommendations 

Most survey respondents understood the correct definition of independence and were satisfied with the 
requirements. Based on these results, the independence objectives in DO-178B should not be changed. 
However, clarification of the meaning of independence and the purpose of the requirements might be 
helpful. 
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2.6 Section F: Modified Condition Decision Coverage 

DO-178B requires modified condition decision coverage (MCDC) for Level A software. This 
requirement has been controversial, with many people publicly claiming that MCDC never or rarely finds 
errors; and, if it does find errors, those errors are not safety-related. Others, however, look at the 
requirement for MCDC as an assurance mechanism to check the adequacy of requirements-based testing. 
From either perspective, the cost of achieving MCDC has been an issue. Because MCDC is only required 
for Level A software, only those respondents with Level A experience were asked to answer the questions 
in this section. 

The following were the objectives for this section. 

• Determine whether people understand MCDC and whether it is done correctly when required 

• Determine if the cost benefit trade is worthwhile in going to MCDC coverage 

• Determine if MCDC is the appropriate stopping criteria for Level A testing 

2.6.1 Significant Findings 

A number of questions were asked to determine opinion about the difficulty of perfonning MCDC and 
about how MCDC was being perfonned. About 60% of the survey respondents indicated they had 
experience with Level A software [F0]. These respondents are referred to as "section F respondents". 
These respondents clearly believed that MCDC was difficult. When asked about the difficulty of meeting 
the MCDC objective [FI], 2% (three people out of 146 answering this question) thought it was trivially 
simple , 19% thought it was moderately simple, 55.5% thought that it was moderately difficult, and 23% 
thought that it was extremely difficult. 

In general, the section F respondents took a number of different approaches to MCDC. About 59% 
indicated that they did requirements-based testing and used the results to determine if additional 
structural testing is necessary, 32.5% said they did structural coverage independent of requirements- 
based testing, while others said that they did the two types of testing in parallel to save time or because 
automated tools do not support a single test suite. About 74% of the respondents performed MCDC on 
source code [F2]; 47% performed it on object code [F3], Thirteen respondents claimed to not perform 
MCDC on either the source or object code. Given that the section F respondents were all supposed to 
have been directly involved in the development or approval of Level A software, the number who had not 
performed MCDC should have been zero. 

Concerning effectiveness [F4], 12% said they never found errors using MCDC, 59% said they rarely 
did, 25% said they occasionally did, and 4% said they frequently did. Some of the notes provided by the 
respondents to this question were interesting. One said that they were surprised to find branches in the 
object code that they were not expecting; they thought they would not have found them doing coverage at 
the source code level. Another thought that knowing they had to meet the MCDC objective had a 
significant, positive influence on the design — "starting to think about structural coverage at verification 
time has little benefit." Another noted that the errors that had been found during MCDC testing were 
generally not the errors that MCDC testing was intended to find. A couple of respondents thought that 
errors found with MCDC would be found by other types of structural testing. Finally, one person noted 
that the errors found by MCDC were rare but they were the hardest ones to find. 

Of the 116 people who responded that they rarely, occasionally, or frequently found errors while 
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performing MCDC, 21 of them said they had found safety-critical errors. Seventy-seven respondents said 
that they had not found safety-related errors, and 18 either did not know or declined to answer [F4B], 
Although it was not clear whether the 21 people corresponded to at least 21 unique discoveries of safety- 
critical errors, this result differs from the common industry perception that MCDC does not find safety- 
critical errors. 

Doing a trace from source to object code as part of MCDC was also addressed. Of the section F 
respondents, 57% said they did traceability from source to object code [F5], Of those who did, 75% said 
additional verification resulted from source to object tracing [F6], In cases where additional verification 
resulted, 41% responded that the verification found additional errors [F7], and 50% of those said they 
found safety-related errors [F7a], 

Many of the complaints about the MCDC requirement concern perceived cost issues. The survey 
posed several questions to obtain MCDC-related cost information. When asked to rate cost attributed to 
performing MCDC, 1% said it was negligible , 16% said it was small, 67.5% said it was substantial, and 
7% said it was nearly prohibitive [F9a], Several noted cost reductions once they started to use supporting 
tools. The time attributed to performing MCDC resulted in similar percentages [F9b], The question 
about which aspects of the MCDC objective are the most difficult [Fib] elicited a large number of 
comments asserting the need for inexpensive tools to assist with the process. 

The survey also asked the respondents to rate the value of MCDC for Level A systems [F9d], The 
survey results showed mixed opinions: 7.5% said it was extremely valuable, 40% said it was somewhat 
valuable, 40% said that it was of marginal value, and 10% said it was of no value. Respondents were also 
asked how much software development time and cost their company would devote to performing MCDC 
if they were not required to meet the objective of DO-178B [F9c], Approximately 5% said same as now, 
19% said almost as much as now, 61% said substantially less than now, and 10% said none. Given the 
frequent complaints about MCDC expressed by many members of the industry, it was interesting that half 
of the respondents stated it was somewhat or extremely valuable, and nearly a quarter of the respondents 
said they would do the same or almost as much as now even if they were not required to do so by DO- 
178B. 

2.6.2 Summary and Recommendations 

In summary, the survey provided evidence that MCDC finds errors, even safety-critical errors. At the 
same time, MCDC was perceived as difficult and expensive. A possible follow-up investigation would be 
to determine whether those who are finding errors using MCDC differ significantly in the way they 
perform it from those who do not find errors. Follow-up investigation also could be perfonned to 
determine if some of the time and resources devoted to MCDC are wasted or applied inefficiently. 

Recommendation FI: The FAA and the industry, in conjunction with the RTCA, should document the 
intent of the MCDC objective and means for achieving MCDC. In addition, a tutorial should be 
developed for perfonning and evaluating MCDC. 

Recommendation F2: The FAA should initiate research to explore cost effective means of achieving 
MCDC or its intent. 
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2.7 Section G: Traceability 


Traceability refers to the practice of tracking or tracing requirements through the development and 
verification process. There are many reasons for traceability, including requirements coverage, regression 
analysis, and change impact analysis. During Workshop I, participants expressed concern about the cost 
of traceability, confusion over the amount of traceability required for different applications, and what 
techniques and evidence of traceability were acceptable to the FAA. Both airborne and ground-based 
respondents answered the questions in this section. 

The following were the objectives for this section. 

• Determine the level of understanding of traceability 

• Determine the cost of the requirements for traceability 

• Determine the perceived value of traceability 

2.7.1 Significant Findings 

Respondents were asked a series of questions to determine their practices regarding traceability. The 
survey results provided strong evidence that traceability was used effectively in most cases. Nearly all of 
the respondents (99%) used traceability for requirements coverage [Gl], 91% used it for regression 
analysis [G3], and 81% used it for change impact analysis [G5], When asked if traceability was done 
only for certification, 77% of respondents responded no and 23% responded yes [G7]. At first glance, a 
response rate of 23% indicating they did traceability only for certification may seem disturbing. 
However, of that 23%, only 2 respondents did not use traceability for requirements coverage, regression 
analysis, or change impact analysis. 

Some misunderstandings about traceability from source to object code were evident in the survey 
responses. Approximately 27% of the respondents indicated that they documented traceability from 
source to object code [Gil], regardless of whether they were developing Level A software. Of the 186 
respondents who did not document traceability from source to object code, 26% of them indicated that 
they always worked on Level A or B software. Responses to the follow-up question [G12] asking for the 
reason traceability from source to object code was documented, were quite mixed. Respondents indicated 
company procedures, ACO requirements, DER requirements, DO-178B requirements, and customer 
requirements as reasons. 

A majority of respondents (55%) said the cost [G13] and time [G14] for traceability was substantial . 
But almost 66% of the respondents said they would do the same or almost as much as they currently do 
without the DO-178B requirement [G15], A full 87% of the respondents indicated that they believed DO- 
178B's requirements for traceability were somewhat or extremely valuable. Based on these results, the 
benefit of traceability appears to be clear. 

2.7.2 Summary and Recommendations 

In general, traceability is used effectively and is well understood. However, there appeared to be 
significant misunderstandings with regard to requirements for establishing traceability from source to 
object code. 

Recommendation Gl: The FAA and the industry, in conjunction with the RTCA, should clarify the 
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intent of DO-178B with respect to source to object code correspondence. The FAA should develop 
policy to standardize the application of source code to object code correspondence. 
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2.8 Section H: Quality Assurance 

Three objectives for the software quality assurance (SQA) process in Annex A of DO-178B are as 
follows. 

SQA Objective 1: Assurance is obtained that software development and integral processes comply 
with approved software plans and standards 

SQA Objective 2: Assurance is obtained that transition criteria for the software life cycle processes 
are satisfied 

SQA Objective 3: Software conformity review is conducted 

Participants at Workshop I questioned the effectiveness of these SQA objectives. In particular, 
workshop participants expressed concern that an applicant could meet all of the SQA objectives without 
producing a quality product. This section of the survey was designed to solicit opinion about the value of 
the SQA objectives, and determine if certification cost could be reduced by lessening quality assurance 
requirements in DO-178B. Attention was focused on the value of each of the three SQA objectives, and 
on the cost and time of compliance. Both airborne and ground-based respondents answered the questions 
in this section. 

The following were the objectives for this section. 

• Determine whether quality assurance as defined in DO-178B is perceived to be useful 

• Detennine the cost of compliance 

2.8.1 Significant Findings 

Respondents provided their opinions on the value of each of the three SQA objectives. The survey 
results showed strong support for quality assurance as it exists in DO-178B as a whole. Approximately 
79% of the respondents rated SQA objective 1 as extremely or somewhat valuable [HI]. One person 
commented "Plans and standards are just paper unless they are followed. Without some tool for 
providing assurance that they were applied they are meaningless... experience has shown that the 
apparent gains of not doing so are extremely short-tenn gains and do not offset the long term costs 
associated with not having them." 

A smaller majority of respondents (57%) stated that SQA objective 2 for transition criteria was 
extremely or somewhat valuable. Only 12% claimed objective 2 was of no value [H2], Similarly, SQA 
objective 3 for software conformity review was also considered valuable. Over 72% of the respondents 
rated it extremely or somewhat valuable and only 4.5% rated it of no value [H3], 

Both the cost and time attributed to complying with the SQA objectives appeared reasonable: 58% of 
the respondents considered them to be small or negligible. At the other extreme, almost 32% stated the 
cost attributed to quality assurance was substantial [H4], An in-depth analysis of the survey results 
revealed a statistically significant correlation between cost of quality assurance and volume of DO-178B 
projects. Respondents from companies that had a high number of approvals attributed less cost to quality 
assurance than respondents from companies with a small number of approvals. One explanation is that 
the resources and infrastructure, and therefore the cost, required for quality assurance was spread over 
several projects. The fact that almost 73% of the respondents would do the same as now or almost as 
much in the area of quality assurance if it were not required by DO-178B further indicated the importance 
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of quality assurance [H6], 

2.8.2 Summary and Recommendations 

The survey results showed strong support for quality assurance as it exists in DO-178B. It was not 
clear whether the requirements could be reduced to save cost without sacrificing quality or safety. 
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2.9 Section I: Documentation 


DO-178B states that the minimum software life cycle data to be submitted to the certification authority 
for software approval is the Plan for Software Aspects of Certification (PSAC), the Software 
Configuration Index, and the Software Accomplishment Summary. In addition, DO-178B states that 
applicants should make available other data or evidence of compliance as requested by the certification 
authority. Section 11 of DO-178B discusses the characteristics, form, and content of the software life 
cycle data. 

Workshop I revealed a widespread belief that ACOs routinely demand software-related documentation 
that cannot be justified by DO-178B or the FARs, and that requirements for packaging documentation are 
unreasonable. There was also a strong presumption that requirements for documentation could be 
deleted, with no adverse effect on safety. Both airborne and ground-based respondents were prompted to 
answer the questions in this section. 

The following were the objectives for this section. 

• Determine if any of the requirements for documentation result in unnecessary rework or write-only 
documents 

• Determine the cost of unnecessary documentation 
2.9.1 Significant Findings 

Survey respondents were asked questions to address both the packaging and content of documentation. 
When asked how closely they follow the document outlines given in Section 11 of DO-178B, 
approximately 73% of the respondents indicated strict or close adherence with some exceptions [II], The 
format of data prepared by applicants was generally driven by company procedures, with only 23% of 
format requirements dictated by ACOs or DERs [12], No responses to this question suggested a problem 
with the formats described in Section 11. Indeed, most replies indicated that the guidance in DO-178B 
was useful rather than a hindrance, citing consistency and ease of comprehension for purposes of review 
and audit. 

To further explore the packaging issue, respondents were asked if they had experienced a rejection 
based on format of certification data within the last five years [13], Only 16% of the valid responses were 
yes. Reasons for rejection differed little from those found elsewhere: confusion between regulatory and 
contractual considerations, personal preferences of ACOs and DERs, and disagreements over 
interpretations [15], Supplemental notes to this question pointed to rejection of electronic data in favor of 
paper copies of documents. One note asserted that personnel changes at the ACO invariably resulted in 
renegotiations of acceptable formats, and that these personnel changes were relatively frequent. 

In addition to concerns about packaging, significant misunderstanding among applicants of what is 
and is not required in the submittal of certification data was evident. When asked for examples of data 
requested by ACOs beyond the requirements of DO-178B or the FARs [16], few of the responses were 
clearly beyond such requirements. More than half of the examples given were within an ACO's 
legitimate purview under current mles [17], Such examples included coding standards, configuration 
index data, and status of problem reports. Of the remaining data, most was tied to contractual or foreign- 
authority considerations (and was thus beyond the scope of this survey ) or had insufficient detail to allow 
a judgment as to the merits of its claims. It was clear that much misunderstanding and misinfonnation 
existed in the recording, retention, and submittal of certification data. Given the misunderstandings about 
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what constituted unreasonable requests for data, it was difficult to interpret responses about associated 
costs [18]. 

The survey addressed unnecessary or unreasonable requirements for rework from several different 
perspectives. Respondents were asked if the FAA had ever requested data or documentation that was not 
specified in the applicant’s PSAC [I8A], The 71% of the respondents who said no suggested that 
applicants and regulators shared common expectations of each other in the majority of projects. On the 
other hand, 55% of the respondents said they were required to update documentation at or near the end of 
a project where the document revisions had no effect on the safety or maintenance of the product [19]. In 
most (61%) of these instances, approval was delayed due to these updates. In this case, the implication 
was that such rework was of little or no value, and caused unnecessary delays in approval. An interesting 
question not addressed by the survey is whether the requirements in DO-178B for updates are clear. 

In a similar vein, there were assertions of documentation solely to meet certification requirements. 
Question 115 asked if data (excluding the PSAC, Configuration Index, and Accomplishment Summary) 
was produced solely to meet certification requirements and not used as part of the software development 
process. Fully 40% of the respondents believed that they produced at least some such data. Although it is 
reasonable to assume that producing data "for show" causes unnecessary costs, the survey uncovered few 
useful clues to provide further insight [116]. Most of the respondents listed data deliverables with no 
explanation as to why the data was thought to be useless, and most of the more elaborate responses 
suggested a misunderstanding of regulatory requirements. 

Finally, questions in this section gauged whether people believed the data requirements of DO-178B 
to be excessive, for both new and derivative equipment [111-14], In both cases, two-thirds of the 
respondents said that the standard’s requirements were not excessive. The responses of the one-third who 
said the requirements were excessive provided no basis for constructive action. 

2.9.2 Summary and Recommendations 

There was significant misunderstanding among applicants of what is and is not required in the 
submittal of certification data. In addition, there was evidence to suggest that the FAA has made 
seemingly unnecessary requests for updates of data at the end of projects and requests for data that is not 
used by applicants. 

Recommendation II: The FAA should make compliance requirements explicit in the correct usage of 
DO-178B’s Section 11 guidelines. 

Recommendation 12: The FAA should make compliance requirements explicit regarding the data 

required for certification. 

Recommendation 13: The FAA should investigate the reason for end-of-project updates to software data 
or documentation that had no impact on the approval or continued safety or maintenance of the product. 
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2.10 Section J: Tool Qualification 


The concept of tool qualification is unique to civilian aviation. Other industries do not typically 
require tool qualification prior to use. Qualification guidance in DO-178B is based on whether the tool is 
a development tool or a verification tool. During Workshop I, participants raised concerns about the 
benefit of tool qualification on the overall quality of the product. In particular, a few participants claimed 
that tool qualification does not find errors. Some workshop participants also expressed concern about the 
clarity of the qualification guidance in DO-178B and questioned why tools with known pedigrees require 
qualification. Only those airborne and ground-based respondents with experience in tool qualification 
were prompted to answer the questions in this section. 

The following were the objectives for this section. 

• Determine whether tool qualification as required by DO-178B provides any benefit with respect to 
product quality 

• Detennine if the cost is justified by the benefit 
2.10.1 Significant Findings 

Approximately 52% of the survey respondents indicated they had experience with tool qualification 
[J1 ]. These respondents are referred to as "section J respondents". The survey results clearly indicated 
that errors had been found in tools during the qualification process. Of the section J respondents, 
approximately 44% found errors with a development tool during tool qualification [J2]; 57% found an 
error with a verification tool during tool qualification [J2A], When asked their opinion about the 
requirements for tool qualification, 59% of the respondents said the requirements for development tools 
were about right, 9% said they were too weak, and 32% said they were too stringent [J3], Similarly, 
about 74% of the section J respondents said that the qualification requirements for verification tools were 
either too weak or about right [J3AA], 

Queries into cost showed that 60% of the respondents considered the cost attributed to tool 
qualification to be small or negligible, 36% considered the cost to be substantial, and 4% considered the 
cost to be prohibitive [J5] . Results for time attributed to tool qualification were comparable: 67% of the 
respondents claimed time to be small or negligible, 30% claimed it to be substantial, and 3% claimed it to 
be nearly prohibitive [J6], Given that tool qualification has found errors and approximately two-thirds 
indicated that cost and time were not substantial, one would expect that companies would continue to 
qualify tools independent of the regulatory requirement. 

There was, however, evidence that respondents believed that some of the resources expended for tool 
qualification were unnecessary. When asked "how much development time and cost would your 
company devote to tool qualification if you were not required to meet the objectives of DO-178B", many 
of the respondents (over 63%) claimed substantially less than now or none [J7], Also, 28% of the section 
J respondents said that some tools should not require qualification, although they were required to do so 
[J4], When asked to explain why some tools should be exempt from qualification requirements, the 
respondents' comments varied [J4A], Several respondents advocated support for COTS tools or tools 
with an industry-wide reputation being exempt from qualification. Other responses indicated possible 
inconsistencies in the requirements for tool qualification. 

Clearly a majority of the respondents believed that tool qualification was useful and the qualification 
requirements were not excessive. One respondent wrote, "Software verification tool qualification is very 
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easy. It includes things that I would expect an engineer to do anyway; know the tool requirements, how 
to test it, and actually test it." Others echoed this opinion. Considerable support (indicated by an 83% 
positive response rate) existed for establishing a national repository for qualified tools [J3A], People 
raised logistical issues including purchasing, licensing, and repository control that should be addressed 
before considering a national repository. 

2.10.2 Summary and Recommendations 

In summary, the survey results showed that the process of tool qualification provided a benefit. 
Respondents considered it valuable, because tool qualification found errors, and the cost and time 
appeared relatively small for a majority of the respondents. There may be benefit, however, in clarifying 
the requirements for tool qualification for both industry and regulators so that requirements are applied 
consistently, and in further investigating the concept of a national tool repository. 

Recommendation Jl: The FAA should clarify compliance requirements and intent for tool qualification. 
In addition, the FAA should clarify the definitions of development and verification tools. 

Recommendation J2: The FAA and the industry should investigate techniques for tool qualification that 
will allow qualification to be faster and cheaper. The FAA should detennine the feasibility of a national 
repository for qualified tools and acceptance criteria for the use of these tools. 
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2.11 Section K: Safety 


Safety is an important issue to systems developers and the FAA. Participants in Workshop I were 
vocal in their concerns about the relationship between DO-178B and safety. At Workshop II, participants 
ranked the safety issue as the third most important issue for further investigation. 

Beliefs differ about the connection between safety and DO-178B. Some argue safety issues are 
adequately addressed through aspects of system development such as the Society of Automotive 
Engineers (SAE) standards ARP-4754, Certification Considerations for Highly-Integrated or Complex 
Aircraft Systems, and ARP-4761, Guidelines and Methods for Conducting the Safety Assessment Process 
on Civil Airborne Systems and Equipment (refs. SAE ARP4754, SAE ARP4761). Others advocate 
greater focus on safety in DO-178B. This section of the survey attempted to gather data on safety 
information relevant to software development, safety-related errors, and derived requirements. Both 
airborne and ground-based respondents were prompted to answer the questions in this section of the 
survey. 

The following were the objectives for this section. 

• Determine whether people think that DO-178B adequately addresses the effect of software on the 
safety of the overall system 

• Determine safety- related activities used in addition to DO-178B 

• Determine how derived requirements are addressed 

2.11.1 Significant Findings 

Several questions focused on industry's view of the adequacy and effectiveness of DO-178B for 
software-related safety issues. When asked whether they believed DO-178B adequately addressed safety 
issues, approximately 59% of the survey respondents indicated they believed it does, while 41% indicated 
they believed it does not [K 1 ]. 

Survey participants answered questions about additional safety activities performed beyond that 
required by DO-178B. Nearly 49% of the respondents asserted that they performed additional activities 
for software safety on quite a few, nearly all, or every project [K2], More than half of the respondents 
(56.5%) gave safety-related information, other than software level, to developers [K6], Care must be 
taken in interpreting these results, however. Answers to the question about the additional safety-related 
activities they performed varied as to what activities were considered software related but outside of DO- 
1786 [K7], Although many noted activities included in the system and safety guidelines, ARP-4754 and 
ARP-4761, the survey still left unclear how much of systems information gathered during these activities 
was used during the software development. 

Of perhaps more use in providing information about the issue, 28% of those surveyed stated that they 
worked on a system that had a software-related system error resulting in a service bulletin or 
Airworthiness Directive [K3], The software for those systems was DO-178B compliant. When asked 
about the source of these errors [K4], 146 causes were noted: 69 were related to requirements, 32 to 
design, and 35 to coding errors. This result was consistent with the ranking of requirements as an 
important issue for research at Workshop II. 

Question K5 asked those respondents who indicated they had not worked on a system with a software- 
related system error their opinion about the contribution of DO-178B to the absence of errors. Somewhat 
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surprisingly, only 37% claimed that DO-178B made a major contribution to the absence of software- 
related system errors; 63% claimed it made no contribution or a minor contribution to the absence of such 
errors [K5], This result may be interpreted as a comment either on the perceived ineffectiveness of DO- 
178B with respect to system errors, or on the perceived unimportance of software on system errors. 

A final objective of this section was to determine how industry treats derived requirements. Derived 
requirements, according to DO-178B, are software design features that cannot be directly traced back to a 
high-level software requirement. DO-178B requires derived requirements to be identified and subjected 
to a system safety assessment. Only 9% of the respondents said that they did precisely this [K8], Other 
techniques for treating derived requirements included: 

• create one or more "place-holder" requirements at a higher-level to which the derived 
requirements can be traced (21%), 

• add rationale to the documentation (16%), 

• find existing and somewhat related requirements at a higher level to which the identified derived 
requirements can be traced (40%), and 

• do not identify derived requirements as such and treat them as an artifact of the design process 
(14.5%). 

Another 2% checked other and specified what they did. Most of these were either very similar to the 
other choices given or reflected that the respondents did several of the other choices, or that the approach 
varied by project. 

Despite the fact that a large majority of respondents handle derived requirements in a manner 
inconsistent with DO-178B, 22.6% of the total number of survey respondents stated that derived 
requirements led to a safety-related modification to the system design [K9], This result attests to the 
importance of derived requirements with respect to safety. 

2.11.2 Summary and Recommendations 

Although the survey did not provide enough information to reach conclusions on the safety issue, the 
results justify further research and information collection. The results also indicated that derived 
requirements are not being handled as specified in DO-178B. 

Recommendation Kl: The FAA should study the relationship between DO-178B and safety and the 
activities actually performed by the industry to ensure system safety. 

Recommendation K2: The FAA should clarify compliance requirements and intent for derived 
requirements. 
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2.12 Section L: The DER System 


This section of the survey was designed to address inconsistencies and inadequacies specific to the 
DER system. This section was divided into two parts: Section L assessed satisfaction with and 

effectiveness of DERs in the airborne world, and Section GL assessed the feasibility for delegation of 
compliance-finding to DERs working on ground-based equipment. 

The DER system is used to augment the federal work force. Under Title 14 Code of Federal 
Regulations (CFR) Part 183, Representatives of the Administrator, the FAA is permitted to delegate some 
findings of compliance to DERs. DERs provide the majority of data approvals for airborne products in 
the United States each year. Hence, the effectiveness and efficiency of the DER system is significant to 
the certification activities of most applicants and to the FAA. 

At Workshop I, some participants claimed that the DER system does not work well for software 
aspects of certification. Some thought that the FAA should delegate more authority to the software DERs 
than they currently do, while others thought that the current qualification standards for DERs are 
inadequate. A recent report from the Government Accounting Office (GAO) on aircraft certification also 
highlighted delegation and qualification issues (ref. GAO). In addition to delegation issues, workshop 
participants raised issues related to DER involvement with TSO projects. The TSO system now routinely 
deals with complex computer-based systems. However, current policy does not give approval authority 
to DERs on TSO projects. Only those airborne respondents with experience working with a software 
DER were prompted to answer the questions in this section. 

The following was the objective for this section. 

• Determine whether there are inadequacies, inconsistencies, and inefficiencies in the DER system 
for airborne software 

2.12.1 Significant Findings 

In this section, 111 airborne respondents indicated they had experience working with a software DER 
[LI]. For purposes of this paper, respondents in this section are referred to as "section L respondents". 
The following subsections explore possible inadequacies in the DER system by examining satisfaction 
and training issues, and explore possible inconsistencies and inefficiencies by examining interaction 
between ACOs and DERs, interaction among multiple DERs, and DER competency. 

2.12.1.1 Inadequacies 

Of the section L respondents, more than 80% said that they were somewhat or very satisfied with their 
primary software DER. Just under 10% were either somewhat or very dissatisfied, leaving few (less than 
8%) neutral [L2], Question L15 sought insight into repeated dissatisfaction, if any, with a given software 
DER. A large majority, 90%, reported no such pattern; only 10% (17 people) reported repeated 
dissatisfaction with their DER. There were two supplemental notes that helped explain this 
dissatisfaction. In one case, the DER in question was new. In the other, the respondent suggested that the 
DER was not familiar with a particular functional discipline. 

In another attempt to gauge overall satisfaction with the DER system, respondents were asked their 
opinion of the degree of delegation to software DERs [L12A], Just over 70% thought the degree of 
delegation was about right, 25% thought that more authority should be delegated to DERs, and the 
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remaining 5% said that DERs had too much authority now. When asked if the degree of delegation to 
software DERs caused adverse effects on costs and schedules [L13], 74% reported no such adverse 
effects, while 26% did. Several of the supplemental notes to this question suggested that more flexibility 
would be useful, with more or less authority delegated to any given DER, depending on individual 
abilities. 

Training for DERs was examined from two perspectives: training provided by the FAA, and training 
provided by companies. Almost 44% of the section L respondents believed that FAA training of DERs 
was inadequate [L21], Some confusion is possible in this area, given that the FAA does not "train" DERs 
in the traditional sense of the word. Rather, the FAA appoints individuals as DERs who already meet 
certain criteria, then expects suitable levels of practical activity and periodic exposure to the agency and 
its various forms of guidance. When asked what training should be provided to DERs by the FAA, many 
respondents suggested standardization of DER behavior at the national level, and a wish for deeper 
technical skills of DERs in software work [L22], 

In contrast, survey results showed that a little over 28% of the respondents answering the training 
question worked for companies that provided training specific to the software DER function [L23], 
Between half and three-quarters of that 28% said that their companies provided training in DER 
procedures, software engineering, the certification process, DO-178B, and system engineering [L24], 
There was roughly uniform coverage of all areas for the companies in question. 

Finally, respondents answered questions about the competency of software DERs. Approximately 
80% of the respondents said the DERs they had worked with had an acceptable background for what they 
were approving [L6]. Of the 20% that reported their DERs had an inadequate background, about two- 
thirds cited lack of software engineering knowledge as a significant shortcoming. About half reported 
lack of DO-178B knowledge in their DER, and a little over a third reported lack of certification knowledge 
[L7], Respondents were also asked whether a software DER had asked anything technical of the 
respondent that was not a legitimate part of the FAA software policy and guidance [LI 2], Policy and 
guidance includes the FARs, DO-178B, FAA Notices, Advisory Circulars and FAA policy memos. Only 
11 respondents said that their DERs had levied requirements beyond the official guidance material such 
that the impact was substantial. 

2.12.1.2 Inconsistencies and Inefficiencies 

The first questions here addressed interaction between ACOs and DERs. Question L3 addressed the 
ability of DERs to provide certification guidance by asking whether the respondent’s software DER had 
been in dispute with or overruled by an ACO within the last 3 years. Only 19 of the section L 
respondents (about 11%) answered yes. Similarly, L4 asked whether an ACO had required, within the 
same period, additional data from the respondent’s project, data not previously identified by the DER. A 
little less than 15% said that the ACO had required additional information. 

Question L17 asked if the respondent's company had ever proposed a delegation to a software DER 
that the ACO rejected. Only 1 17 of the section L respondents answered this question. Of those, 86% said 
no. The 14% who said yes had a variety of explanations for the rejection [LI 8], The main reasons had to 
do with ACO familiarity with the DER in question and a preference by the ACO to retain direct 
involvement in the projects. 

Another issue regarding interaction between ACOs and DERs involved compliance findings for TSO 
projects. Respondents were asked if they had worked on any TSO project where software DERs were 
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used to approve the data [L10], More than half (53%) of those responding to this question said yes. The 
high frequency suggests that the FAA should investigate the appropriateness of using DERs on TSO 
projects. 

Respondents were asked about their experience with interactions among DERs on large projects. 
Specifically, respondents were asked if they had worked in recent years on a project having multiple 
software DERs with overlapping responsibilities, and 38% said yes [L8], Of the 58 responses in the yes 
category, 41% contended that disagreements among the multiple DERs led to schedule delays and wasted 
resources [L9], The remaining 59% believed that no delays or resource waste was attributable to 
disagreements among the DERs. Question Lll asked about a succession of DERs, rather than those 
whose activities overlap. In this case, 15% of the respondents said they had suffered major rework as a 
result of a new DER disagreeing with his or her predecessor. The rest said no. One supplemental note 
admitted to some disagreements but said that no major rework ensued. 

2.12.2 Summary and Recommendations 

Overall, there was a high level of satisfaction with the DER system. The survey indicated additional 
training, especially in software engineering and DO-178B, is needed. However, there appeared to be 
some misunderstanding about who is responsible for providing DER training. Survey results also 
indicated that software DERs were being used to approve data on TSO projects. 

Recommendation LI: The FAA should evaluate the adequacy of their current criteria for selecting 
software DERs and the procedures to ensure that the DERs meet the competency levels. 

Recommendation L2: The FAA should investigate whether software DERs should participate officially 
in findings of compliance on TSO projects. 
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2.13 Section GL: Using DERs for Ground-based Systems 

For ground-based systems, there is no DER-equivalent function specified in the FARs. Although 
DERs participate in some ground-based development projects, approval of design data for these systems 
currently remains with the FAA directly. 

This section of the survey investigated whether the DER system, as implemented in the airborne 
sector, would be appropriate for ground-based systems, too. Software DERs assist in ground-based 
development projects, but do not have authority to approve design data. Workshop I participants 
speculated that cost and time for approval could be reduced if the authority in Title 14 CFR Part 183 were 
expanded to allow DER approval of data for ground-based systems. To address this issue, the survey 
posed questions about DER contributions to ground-based projects. Only ground-based respondents who 
had worked with DERs answered the questions in this section. 

The following was the objective for this section. 

• Determine whether the DER system would be appropriate for ground-based systems 

2.13.1 Significant Findings 

Of the 86 ground-based respondents in this survey, only 30 reported experience working with a 
software DER on a ground-based project [GL1], These 30 are referred to as "section GL respondents". A 
majority (21) of the section GL respondents worked with navigation systems. 

The survey results showed that software DERs were used on ground-based systems for reviewing 
architecture, allocating software assurance levels, setting up DO-178B compliant development processes, 
and reviewing submissions of DO-178B data prior to FAA submission [GL2J. To a lesser extent, DERs 
either conducted or provided oversight to reverse-engineering activities and conducted audit activities 
such as test witnessing and traceability thread analysis. 

Twenty-eight of the 30 section GL respondents indicated that software DERs helped improve the 
ability to understand and comply with DO-178B [GL4], Only 8 of 30 respondents indicated that the 
software DER reduced the effort involved in making improvements to the software architecture [GL5], A 
majority (18) reported that the software DER helped reduce the delay in approval of submissions to the 
FAA [GL6], 

Overall, the data suggested a positive impact of DER involvement. A majority of the section GL 
respondents was satisfied with their experience with software DERs, with 63% reporting they were very 
satisfied , and an additional 30% reporting they were somewhat satisfied [GL3], Two comments on this 
question, however, indicated that a lack of consistency between DERs was observed on ground-based 
equipment. Over two-thirds of the section GL respondents indicated that the software DER authority 
under Title 14 CFR Part 183 should be expanded to allow DERs to attest to the compliance of a submittal 
[GL7], 

2.13.2 Summary and Recommendations 

The survey results showed that respondents believed that DER involvement in ground-based 
development projects had a positive impact, in particular in understanding DO-178B and in reducing the 
time to approval. Survey respondents favored modification of Title 14 CFR Part 183 to allow DERs to 
participate fully in ground-based projects, with assurance that findings and recommendations of the 
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airborne community are considered. 

Recommendation GL1: The FAA should investigate expansion of Title 14 CFR Part 183, 
Representatives of the Administrator , to allow designee authorization for the ground-based community. 
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2.14 Section M: Availability of Information about the Certification Process 

A claim raised by attendees of the workshops was that insufficient infonnation is available about the 
certification process. If this is correct, it has the potential for affecting certification costs significantly. 

The following were the objectives for this section. 

• Detennine if the necessary information about software aspects of the certification process is 
readily available and, if not, what information is missing 

• Determine desired fonnat(s) for future information 

2.14.1 Significant Findings 

Section M began with a general question about obtaining information on software aspects of 
certification [Ml], The text of the question asked respondents to what extent they agreed or disagreed 
with the statement "Sufficient infonnation is available for obtaining software approvals." The response 
choices were strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, and 
strongly disagree. 

The survey responses to this question were encouraging: 72.3% of the respondents indicated a 

positive or neutral response. The positive interpretation of the responses to this question was fairly 
consistent with the responses to other, more detailed questions in this section. Note, however, that even 
with this favorable response, 27.3% of respondents either somewhat disagree (20.6%) or strongly 
disagree (7.1%) — indicating room for improvement. 

Survey questions MIA through M7A asked respondents to provide their opinion about the availability 
of infonnation in eight specific areas. The areas were: interpreting DO-178B [MIA]; establishing 
software levels [M2]; coordinating with the approving authority [M3]; preparing for audits [M4]; 
obtaining final approval [M5]; obtaining TSO approvals [M6]; obtaining PMA approvals [M7]; and 
resolving open software issues [M7A[. 

In all of these specific areas except one, the responses were similar to those received for the general 
question Ml. Responses that were either neutral or positive ranged from 58.2% to 80.4% in seven of the 
eight cases. The exception was the question that asked the respondents to comment on the statement 
"Sufficient information is available for interpreting DO-178B" [MIA], Thirty-three percent responded 
somewhat disagree and 19.3% responded strongly disagree — a total of 52.3% negative responses. 

The second objective of section M was to detennine desired fonnat(s) for future infonnation [M9 - 
M10], The overwhelming response was a preference for a web site for infonnation distribution. This 
supports the Task Force 4 finding of a need for an electronic repository for regulatory infonnation and 
technical lessons learned (ref. Final, page 84). CD-ROM and diskettes were next in preference. Paper 
was the least preferred media for documentation. 

2.14.2 Summary and Recommendations 

In general, the survey results showed that sufficient infonnation was available about most software 
aspects of certification. However, a large percentage of respondents stated that sufficient information was 
not available specifically about interpreting DO-178B. 
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Recommendation Ml: The FAA should develop a mechanism for providing better info rmation on the 
intent, interpretation, and application of DO-178B. 

Recommendation M2: The FAA should continue efforts to make software-related documentation and 
training materials available on the web. 
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2.15 Section P: Appropriateness of DO-178B for Ground-based Systems 

A great deal of folklore asserts that ground-based aviation equipment is different from airborne 
equipment, and that the methods used for safety analysis and design assurance in avionics cannot or 
should not be applied to ground-based systems. The FAA has the legal and administrative mechanisms to 
ensure that its oversight of airborne equipment manufacturers is independent of manufacturers’ cost and 
schedule pressures. Certification for airborne equipment is accomplished in compliance with the FARs. 
For ground-based systems, however, the developing group has traditionally been responsible for 
commissioning as well. The approval process for ground-based equipment is determined only by FAA 
contract documents, policies, and procedures — not by the FARs. 

Concerns have been raised about the suitability of continuing the current approach for approval of 
ground-based systems and specifically about the applicability of airborne system certification standards to 
ground-based systems. Some have advocated a certification system similar to that used for airborne 
equipment. According to the Task Force 4 report, the Europeans have already begun this transformation 
in their approach to ground-based system approval (ref. Final, page 23). 

Unlike the other sections of the survey that were motivated by issues raised at Workshop I, the 
questions on the appropriateness of DO-178B for ground-based systems were added at the request of the 
FAA. Only ground-based respondents were prompted to answer the questions in this section. 

The following were the objectives for this section. 

• Determine if there is a perception that airborne and ground-based systems are really different 

• Detennine whether DO-178B fits for ground-based systems 

2.15.1 Significant Findings 

When asked whether DO-178B is appropriate for ground-based systems, 85% of the 66 respondents 
that answered this question either thought DO-178B was acceptable for ground-based systems as is, or 
would require only minor changes [PI]. Those who believed it to be inappropriate were asked specific 
questions about the suitability of DO-178B for project planning, requirements definition, design capture, 
coding, verification, and coordination of commissioning [P2a-f], In each of those areas, only a small 
number (3-6 respondents) indicated that DO-178B was not appropriate. 

In addition to the general support for using DO-178B, one-half of the ground-based respondents said 
that the ACO model should be used for commissioning of ground-based systems [P3], That is, the 
responsibility for development and acquisition of a new system should be separated from the 
responsibility of approving that system. It is interesting to note that a significant number of the ground- 
based respondents (37 of 86) indicated that they either did not know or declined to answer this question. 

2.15.2 Summary and Recommendations 

Although the size of the subset of the survey population responding in this section was quite small, the 
survey results showed definite support for the use of DO-178B for ground-based systems. The survey 
results also showed support for the separation of acquisition and approval responsibilities for ground- 
based systems. 
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Recommendation PI: The FAA should investigate the feasibility of implementing a regulatory 
authority independent of the acquisition body for ground-based systems. 

Recommendation P2: The FAA should implement a plan to phase in compliance with DO-178B for 
ground-based systems. 
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3. General Observations 


After careful review of the technical recommendations in the previous section, the technical team 
made the following six general observations. 


Observation 1: Aircraft Certification Offices and other approving authorities create unnecessary cost 

burdens through inconsistent guidance, interpretation, and procedural requirements 
for software-related issues. 


Observation 2: Inconsistencies exist between the airborne and ground-based software approval 

processes that create inefficiencies resulting in added costs for the industry and 
potentially for the FAA. 

Observation 3: The FAA is not keeping pace with rapid advances in software technology, thereby 

delaying the use of potentially cost saving technology. 


Observation 4: The FAA has not allocated enough people with the requisite software engineering 

expertise and knowledge of DO-178B to software approval issues. 

Observation 5: Software issues exist for which FAA software policy or guidance is inadequate. 

Observation 6: Knowledge of and experience with DO-178B varies substantially within the industry. 


These observations are interdependent, with complex cause and effect relationships. All of the 
observations involve in one way or another the interpretation of DO-178B. As anyone who has read 
Section 2 of this report will realize, inconsistent interpretation of DO-178B was a conspicuous theme 
throughout the survey results. 

DO-178B requires interpretation for the following reasons: 


• DO-178B states what is required and not how to develop software; 

• DO-178B has ambiguous terminology; and, 

• DO-178B requires non-traditional software engineering practices, such as MCDC and tool 
qualification, without providing sufficient background information. 

In software plans, companies must propose methods on how they will comply with DO-178B. 
Different companies propose different methods, and these methods may or may not meet the DO-178B 
objectives. Assessment of compliance requires a good software engineering background, because many 
company methods may not consist of traditional textbook approaches. Therefore, the approving authority 
must have an adequate software engineering background to assess compliance. 

Variability in the skill set of some developers and regulators has always existed; why is this an issue 
now? The reason is that the implications of poor software engineering skills and inadequate resources on 
cost and safety have changed with time. Under DO-178A, software engineering skills were less of an 
issue because the amount and criticality of software on the aircraft were relatively insignificant. This 
limited dependence on software under DO-178A was reasonably balanced with the design and assurance 
capability within the FAA and the industry. 

Over the last decade, however, software has moved rapidly from an assistance role to a dependency 
role. Software-based systems monitor and adjust hydraulic, electrical, fuel, and braking systems, and 
control aircraft mdders, wings, and engines. Software is now a factor in safety of flight for fly-by-wire 
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aircraft. RTCA developed DO-178B to meet the rising need for design assurance in software. 

As software dependency in the aircraft increases, there should be similar increases in software 
capability with respect to DO-178B within industry, and increases in FAA capability to assess compliance 
to offset potential for decreased safety or increased cost. The amount and criticality of avionics software 
have increased significantly since the inception of DO-178B. Survey findings, coupled with those of the 
GAO and RTCA Task Force 4, show that industry and the FAA, in general, have not risen to the occasion 
to meet the current need. Knowledge of DO-178B varies substantially among civil aviation developers, 
the capability of the FAA to address software issues remains inadequate, and software policy and 
guidance are lacking. This in turn increases cost and the potential for decreased safety. 

To address the imbalance caused by growing dependence on software, FAA technical resources need 
to be in a position to evaluate and embrace effective certification practices and mechanisms. The FAA 
must acquire and maintain both expertise in software engineering and experience in evaluating new and 
emerging technologies. Timely software policy and guidance is needed. Mechanisms that allow the FAA 
to efficiently manage the approval of all software-based systems need to be established and assessed 
regularly. The aviation industry, including DERs, must also acquire and maintain detailed expertise in 
DO-178B. 
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4. Recommendations 


In response to the six general observations, the technical team developed ten recommendations to 
improve the software approval process. These recommendations are more organizational in nature than 
the technical recommendations given in Section 2. Implementing these recommendations will require 
management support from several FAA organizations. Appendix B contains a letter from the FAA to the 
technical team stating the FAA's commitment to review and comment on these recommendations. 

The recommendations fall roughly into three categories: (1) recommendations to improve consistency 
among approving authorities, (2) recommendations to improve knowledge and skills related to DO-178B 
within both the FAA and the industry, and (3) recommendations to improve the ability of the FAA to 
assure software in a timely manner. 

Recommendations to improve consistency 

• The FAA should determine the causes for the inconsistencies noted throughout the report, and 
detennine what actions, if any, are required in addition to those recommended in this report. 

• The FAA should develop unified policy and guidance for approving software aspects of all aviation 
systems (e.g., airborne, ground-based, and satellite systems). 

• The FAA should institute a regulatory authority independent of acquisition authority for approval of 
ground-based systems. 

Recommendations to improve knowledge and skills related to DO-178B 

• The FAA should hire a sufficient number of software engineering experts to understand the safety 
impact of software technologies for all aviation systems (e.g., airborne, ground-based, and satellite 
systems). 

• The FAA should improve software expertise within the agency by: 

- identifying the minimum software staffing needed to assure a consistent approach and timely 
response for software approvals for all applicants; 

- continually assessing software personnel needs and hiring to meet these needs; 

- requiring software engineers who appoint and advise software designees and who approve 
software to meet at least the same qualifications as the designees; and, 

- creating, funding, and filling software engineering positions throughout the FAA. As a minimum, 
the FAA should fund and fill the software positions to assist the field offices and software 
engineering experts. 

• The FAA should make DO-178B training available to designees. 

• The FAA should require companies providing software for all aviation systems (e.g., airborne, 
ground-based, and satellite systems) to demonstrate competence in DO-178B. Tire FAA should use 
DO-178B competence demonstrated by the applicant as a factor in establishing level of involvement 
in software assessment activities. 
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Recommendations to improve the ability of the FAA to assure software in a timely manner 


• The FAA should establish processes for regularly assessing software policy and guidance needs; 
developing new software policy and guidance when needed; and assessing and enhancing the clarity, 
consistency, and completeness of software policy and guidance. 

• The FAA should establish a means to ensure that the software approval process allows applicants to 
use appropriate new software technologies in a timely manner. 

• The FAA should initiate a program of proactive research to evaluate the potential impact of software 
technology on cost and safety. The research output should influence the development of policy, 
guidance, regulations, and training for software engineering. 


The technical team believes that following these recommendations will promote an efficient software 
approval process, one that can meet the challenges of increased dependence on software in aviation 
equipment. The technical team also recognizes that barriers exist in the current government system that 
constrain the FAA's ability to change rapidly. Nevertheless, both the recommendations above and 
technical recommendations given in Section 2 should be followed to achieve maximum benefit. 
Following the technical recommendations alone would be ineffective. 
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5. Conclusions and Future Plans 


Software has become ubiquitous. As the medium of choice for enabling advanced automation in both 
airborne and ground-based systems, it will form the cornerstone of the National Airspace System. In the 
October 1996 issue of Avionics Magazine, David W. Robb wrote (ref. Robb): 

"It is no secret that aircraft are becoming ever more dependent on their onboard 
electronics. The emerging world of CNS and Free Flight avionics promises to 
accelerate this trend dramatically. As this equipment grows more capable and 
sophisticated, so does the challenge of testing and maintaining it — for the largest 
airline to the smallest General Aviation avionics shop." 

Previous studies by the GAO and RTCA concluded that the FAA cannot effectively oversee the 
current certification process, much less efficiently acquire and regulate new technologies, without making 
significant organizational changes. Evidence gathered from the SSAC workshops and survey clearly 
supports the need for organizational change within the FAA, but it also supports change within the 
aviation software industry as well. The FAA should implement appropriate infrastructure, staffing, and 
policy and guidance to efficiently manage the acquisition and approval of all software-based systems. In 
turn, the industry should continue to work with the FAA to develop new standards for emerging 
technologies and continue efforts to improve software engineering processes. 

In conclusion, the technical team hopes that the preliminary findings and recommendations presented 
here will help the FAA and industry begin to improve significantly the software approval process for 
both. Following the recommendations in this report, along with following-up certain survey findings and 
investigating other software issues not covered by the survey, should go a long way towards ensuring that 
needed improvements will be made. With apologies to Sir Winston Churchill, this report is not the end, it 
is not even the beginning of the end, but, perhaps, it is the end of the beginning. 
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Appendix A 

SSAC Survey Questionnaire 


The complete survey instrument used for the SSAC survey is included here as a reference for the 
discussion on significant findings from the survey. The survey was distributed in a Computer Assisted 
Self-administered Interviewing fomiat on a floppy disk that could be run on any IBM compatible personal 
computer. Given that several paths can be taken through the survey depending on responses to key 
questions, no individual survey respondent likely saw the full set of survey questions. Some of the coding 
instructions for implementing the survey are included to help clarify the paths through the survey. 

The survey was approved for distribution by the Office of Management and Budget (OMB) in 
compliance with the Paperwork Reduction Act of 1995. The Paperwork Reduction Act establishes 
policies and procedures for controlling paperwork burdens imposed by Federal agencies on the public. 
Several goals of the legislation are to: 

• minimize the Federal paperwork burden on individuals, small businesses, and State and local 
governments; 

• maximize the usefulness of infonnation collected by the Federal government; and 

• minimize the cost to the Federal government of collecting, maintaining, using, and distributing 
information. 

Federal agencies, including NASA and FAA, are prohibited from enforcing paperwork requirements 
that are not approved by the OMB or that do not display the OMB approval number. The OMB number 
signifying approval and the statement of compliance with the Paperwork Reduction Act are included in 
the introduction to the survey. 
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Welcome to the 

Survey on Software Aspects of Certification 

Conducted by 
Center for Survey Research 
at 

University of Virginia 


This questionnaire is about your experiences and opinions concerning software aspects of certification. 
In November 1997, a technical team of software engineering and safety experts was asked to study the 
processes by which software developed for airborne and ground-based systems are certified or approved 
by the Federal Aviation Administration (FA A). The team’s objective is to determine if there are ways to 
streamline these processes while maintaining or enhancing their contribution to safety. 

You have been identified as someone who is involved in software development or certification for 
aviation systems. Whether you are new in the industry or an experienced hand, whether you work in a 
large or small firm, your viewpoint is important to this study. We need your responses to ensure that our 
sample of people in the industry is complete and representative. 

By finding out the experiences and opinions of those like you, the team will be able to establish which 
issues and problems are broadly experienced, and what opinions are prevalent in the industry. The 
technical team will be using the results of this survey to detennine if there are ways in which the software 
aspects of the certification process can be made easier and more effective without compromising safety. 
Your ideas for improving the current system are vital to the success of their mission. 

Please answer the questions below candidly and fairly. We assure you that your responses are 
confidential. Neither you nor your company will be identified in any report by the team; results will be 
reported in aggregate form only. 

This survey was approved by the Office of Management and Budget (OMB) in compliance with 
the Paperwork Reduction Act. 


OMB Control # 2120-0635 

The Paperwork Reduction Act: A number of concerns have been raised by the aviation industry about 
the cost and time burden associated with the current software approval process and with the standards 
document, RTCA/DO-178B guidelines "Software Considerations in Airborne Systems and Equipment 
Certification." In response to these concerns, the FAA started the Streamlining Software Aspects of 
Certification (SSAC) program to identify and eliminate unnecessary costs in software aspects of 
certification for both airborne and ground-based systems. The information collection will be used by the 
SSAC Technical Team to assess the extent and significance of issues pertaining to the software approval 
process and to provide an empirical basis from which decisions about software aspects of certification can 
be made. It is estimated that it will take 35 minutes per respondent to complete the survey. The 
collection of infonnation is voluntary, and confidentiality is neither provided nor necessary. The 
information collection will be conducted anonymously. It should be noted that an agency may not 
conduct or sponsor, and a person is not required to respond to a collection of infonnation unless it 
displays a currently valid OMB control number. The paperwork burden associated with this form is 
currently approved under OMB control number, 2120-0635. 
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SSAC Survey - section a 






SSAC Survey - section a 


A3g. Select the highest level of software data you can approve or recommend for approval. 

1 Level A 4 Level D 

2 Level B 9 don’t know/decline to answer 

3 Level C 

A4. In the past five years, have you been directly involved in either the development or approval of software for 
application to airborne systems? 

1 yes 2 no 9 don’t know/decline to answer 

A5. In the past five years, have you been directly involved in either the development or approval of software for 
application to ground-based systems? 

1 yes 2 no 9 don’t know/decline to answer 

** RESPONDENTS WITH GROUND-BASED EXPERIENCE ("yes" on A5) WILL BE TREATED AS GROUND- 
BASED ONLY. 

A6. What type of company do you work for? 

1 an airborne-equipment supplier 4 other [please specify:] 

2 an aircraft manufacturer 9 don’t know/decline to answer 

3 a ground-systems equipment supplier 
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SSAC Survey - section b 


SECTION B - YOUR SOFTWARE TEAM & COMPANY 

I-Bl. Many of the questions that follow refer to “your experience” in software development and approval. In 
answering these questions, please consider your own direct experiences with people and offices as well as events 
that have happened on projects you have worked on. For example, if you know for certain that one of your projects 
was delayed by a problem at another office, you should consider that delay to be part of “your experience,” even if 
you never personally interacted with people in the other office. On the other hand, we ask you not to consider as 
“your experience” the good or bad events you may have heard about from professional colleagues, unless you were 
part of the project, too. By keeping this definition in mind, you will help us to maintain the accuracy of the sun’ey 
results. 

B2. Which of the following best indicates the number of non-clerical people working on your software 
development team? (A team would typically include one or more software engineers, software engineering 
leads, project managers, and a certification liaison.) 


1 

5 people or fewer 

6 

76-100 

2 

6-15 

7 

101 people or more 

3 

16-30 

8 

not applicable 

4 

31-50 

9 

don’t know/decline to answer 

5 

51-75 




If airborne ("yes" to A4 and "no" to A5) complete Bd, otherwise skip to B6 


B3. 

For which of the following airborne systems products have you developed or approved software over the last 


five years? Select all answers that apply. 
1 autopilot 

8 

anti-skid braking 


2 flight management computer 

9 

fuel systems 


3 engine control 

10 

electrical/power systems 


4 passenger entertainment 

11 

communication equipment 


5 displays 

12 

audio systems 


6 navigation equipment 

13 

other [please specify:] 


7 flight controls 

14 

no more/don’t know/decline to answer 

B4. 

Please indicate the aircraft or engine type most typically 

associated with installation of product line(s) you 


work on. Select all answers that apply. 




1 transport aircraft 

7 

small turbine engine 


2 regional aircraft 

8 

piston engine 


3 business aircraft (Part 91) 

9 

other [please specify:] 


4 general aviation aircraft 

10 

not applicable 


5 rotorcraft 

6 large turbine engine 

11 

no more/don’t know/decline to answer 

B5. 

Based on your knowledge, what percentage of the projects you work on receive a Technical Standard Order 


authorization? 




1 None 

5 

51-75% 


2 1-10% 

6 

76-100% 


3 11-25% 

4 26-50% 

9 

don’t know/decline to answer 
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SSAC Survey - section b 


If ground-based ("yes" to A5 ) complete B6, otherwise skip to B7 

B6. For which of the following ground-based systems products have you developed or approved software over 
the last five years? Select all answers that apply. 

1 communications 

2 navigation 

3 surveillance 

4 automation 

B7. Which of the following best describes your own level of experience with developing or approving software in 
compliance with DO-178B, either for airborne or ground-based systems? 

1 I haven’t had any experience with DO-178B. 

2 I’m working on my first DO-178B project. 

3 I’ve completed 1-5 DO-178B projects that have been approved. 

4 I’ve completed 6-20 DO-178B projects that have been approved. 

5 I’ve completed 21-50 DO-178B projects that have been approved. 

6 I’ve completed over 50 DO-178B projects that have been approved. 

9 don’t know/decline to answer 

B8 Please indicate if you have personally been involved with developing or approving software under any of the 
following in addition to DO-178B. 


B8a. 

DO-178A 




1 yes 

2 no 

9 don’t know/decline to answer 

B8b. 

DO-178 




1 yes 

2 no 

9 don’t know/decline to answer 

B8c. 

pre-DO- 178 




1 yes 

2 no 

9 don’t know/decline to answer 

B8d. 

military standards 




1 yes 

2 no 

9 don’t know/decline to answer 

B8e. 

other standards 




1 yes 

2 no 

9 don’t know/decline to answer 


B9. And what about your software development team? In the past five years, has your team been involved with 
developing or approving software under any of the following hi addition to DO-178B? 

B9a. DO-178A 

1 yes 2 no 9 don’t know/decline to answer 

B9b. DO-178 

1 yes 2 no 9 don’t know/decline to answer 

B9c. pre-DO- 178 

1 yes 2 no 9 don’t know/decline to answer 


5 air traffic management 

6 other [please specify:] 

9 no more/don’t know/decline to answer 
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SSAC Survey - section b 


B9d. military standards 

1 yes 2 no 9 don’t know/decline to answer 

B9e. other standards 

1 yes 2 no 9 don’t know/decline to answer 

BIO. As you know, Level A and Level B are the DO-178B levels of software that are the most critical for safety. 
Over the last five years, to what extent have you been involved with development or approval of software that 
has significant sections of code that are considered to be at Levels A or B? 

1 We do not deal with Level A or Level B code. 

2 We deal with Level A or Level B code on only a small portion of our projects. 

3 We deal with Level A or Level B code regularly, but on fewer than half our projects. 

4 We deal with Level A or Level B code on most of our projects. 

5 We deal with Level A or Level B code on every project. 

9 don’t know/decline to answer 

B1 1. If you could put all of the DO-178B software products you have worked on over the past 5 years into a single 
software product, using “kslocs” (thousands of non-commented source lines) estimate the percentage of the 
software in each of the following categories: 

(Enter “999” for Level A if you are unable to estimate the percentage of software hi each category) 

1 Level A 

2 Level B 

3 Level C 

4 Level D 

5 Level E 

SUM = 100% 

B12. Think about the largest software project that you are currently working on. The project could be a piece of a 
product, a single product (such as a Flight Management System), or a combined suite of products (such as an 
Information Management System that contains a flight management system, displays, communication, etc.). 
Please indicate the overall size of the project (measured in “ksloc” — thousands of non-commented source 
lines). 

1 less than 1 ksloc 

2 lk to 9k 

3 10k to 29k 

4 30k to 59k 

5 60k to 99k 

B13. Based on your knowledge, please estimate the number of DO-178B approved projects (including re- 
certifications) your company completes per year. 


1 

None 

5 

21-35 

2 

1-5 

6 

36-50 

3 

6-10 

7 

more than 50 

4 

11-20 

9 

don’t know/decline to answer 


6 100k to 199k 

7 200k to 349k 

8 more than 350k 

9 don’t know/decline to answer 
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SSAC Survey - section b 


If airborne ("yes" to A4 and "no" to A5) complete B14 - B14B, otherwise skip to B15 

B14. For airborne software, which Aircraft Certification Office do you interact with most frequently? (We will 
refer to this ACO as your primary ACO.) 

1 Anchorage Aircraft Certification Office (ACE-1 15N) 

2 Atlanta Aircraft Certification Office (ACE- 1 15 A) 

3 Boston Aircraft Certification Office (ANE-150) 

4 Chicago Aircraft Certification Office (ACE-1 15C) 

5 Denver Aircraft Certification Office (ANM-100D) 

6 Engine Certification Office (ANE-140) 

7 Fort Worth Airplane Certification Office (ASW-150) 

8 Fort Worth Rotorcraft Certification Office (ASW-170) 

9 Fort Worth Special Certification Office (ASW-190) 

10 Los Angeles Aircraft Certification Office (ANM-100L) 

1 1 New York Aircraft Certification Office ( ANE-170) 

12 Seattle Aircraft Certification Office (ANM-100S) 

13 Wichita Aircraft Certification Office (ACE-1 15W) 

14 Brussels Aircraft Certification Division (AEU-100) 

15 not applicable to my company 

16 don’t know/decline to answer 

B14A.Is this the ACO that covers your geographic location ? 

I yes 2 no 9 don’t know/decline to answer 

B14B. Are there other ACOs that you interact with? If so, please select them from this list. Select all answers that 
apply. 

1 Anchorage Aircraft Certification Office (ACE-1 15N) 

2 Atlanta Aircraft Certification Office (ACE-1 15 A) 

3 Boston Aircraft Certification Office (ANE-150) 

4 Chicago Aircraft Certification Office (ACE-l 15C) 

5 Denver Aircraft Certification Office (ANM-100D) 

6 Engine Certification Office (ANE-140) 

7 Fort Worth Airplane Certification Office (ASW-150) 

8 Fort Worth Rotorcraft Certification Office (ASW-170) 

9 Fort Worth Special Certification Office (ASW-190) 

10 Los Angeles Aircraft Certification Office (ANM-100L) 

I I New York Aircraft Certification Office (ANE-170) 

12 Seattle Aircraft Certification Office (ANM-100S) 

13 Wichita Aircraft Certification Office (ACE-l 15W) 

14 Brussels Aircraft Certification Division (AEU-100) 

15 no more/don’t know/decline to answer 

If ground based ("yes" to A5) complete B15, otherwise skip to B16 

B15. Which FAA organization acts as the approval authority for your projects? Note that more than one approval 
authority may be indicated if you are responding for more than one type of project. 

1 Product Team for the Global Positioning System (FAA AND-730) for the Local Area Augmentation 
System (LAAS) and the Wide Area Augmentation System (WAAS) 

2 Product Team for Navigation and Landing (FAA AND-740) for the Instrument Landing System (ILS) & 
Transponder Landing System (TLS) 

3 GPS, Navigation and Landing Branch (FAA AOS-240) for the Differential Global Positioning Systems 
for Special Category I (SCAT-I) 

4 other [please specify:] 

5 not applicable to my company 
9 don’t know/decline to answer 
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SECTION C - FAA POLICY, GUIDANCE, AND AUDITS 

C 1 . Overall, how satisfied are you with the current FAA process for software aspects of certification? Are you: 

1 very satisfied 4 somewhat dissatisfied 

2 somewhat satisfied 5 very dissatisfied 

3 neither satisfied nor dissatisfied 9 don’t know/decline to answer 

I-C2. The following 16 questions are concerned with the extent to which written existing FAA software policy and 
guidance provide the information you need to develop software to meet objectives in DO-178B. 

For each of 16 different subjects, you will be asked whether the written FAA software policy and guidance is ample, 
sufficient, or insufficient, or whether you are unable to rate the level of information provided. 

[DEFINITION: “Written policy and guidance” refers to: the Federal Aviation Regulations, DO-178B, FAA 

Notices, FAA Orders, Advisory Circulars, and FAA policy memos.] 

C2a. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

planning 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2b. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

requirements 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2c. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

design 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don't know 

C2d. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

code 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2e. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

verification 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don't know 
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C2f. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

configuration management 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2g. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

quality assurance 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2h. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

certification liaison 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2i. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

COTS 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2j. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

legacy software and software reuse 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C21. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

user-modifiable software 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2m. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the infonnation you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

partitioning 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2n. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

service history 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don't know 
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C2o. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

tool qualification 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2p. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

classification of major and minor software changes 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2q. In the area below, would you say the written FAA software policy and guidance is ample, barely sufficient, or 
insufficient in providing the information you need to comply with DO-178B; or are you unable to rate the 
level of information provided? 

alternate means of compliance 

1 ample 2 barely sufficient 3 insufficient 4 unable to rate 9 don’t know 

C2r. Is there any area not listed above for which insufficient information is available for you to get your software 
through the approval process? 

1 no 2 yes [please specify:] 9 don’t know/decline to answer 

C2A. Overall, how satisfied are you with existing written FAA software policy and guidance? 

1 very satisfied 4 somewhat dissatisfied 

2 somewhat satisfied 5 very dissatisfied 

3 neither satisfied nor dissatisfied 9 don’t know/decline to answer 


**THIS VERSION OF QUESTION C3 IS FOR AIRBORNE ONLY: 

C3. In the past five years, have you participated in a domestic regulatory audit for software; that is, a software 
review conducted specifically by FAA regulatory personnel (as opposed to a software review conducted by a 
DER)? 

1 yes 2 no 9 don’t know/decline to answer 


**THIS VERSION OF QUESTION C3 IS FOR GROUND-BASED ONLY: 

GC3. In the past five years, have you participated in a domestic regulatory software audit; that is, a software review 
conducted specifically by regulatory personnel such as FAA personnel from the Product Team for the Global 
Positioning System (FAA AND-730), the Product Team for Navigation and Landing (FAA AND-740), or the 
GPS, Navigation and Landing Branch (FAA AOS-240)? 

1 yes 2 no 9 don’t know/decline to answer 


** WORDING IN C4 WILL VARY AUTOMATICALLY DEPENDING ON AIRBORNE OR GROUND-BASED 
RESPONDENT. 


If experienced with audits (1 to C3or GC3) complete C4, otherwise skip to section D 

C4. How many times does [your primary ACO/the approving authority] typically conduct a regulatory software 
audit on each of your projects? 


1 none 

2 once 

3 twice 

4 three times 


5 four or more times 

6 the number of audits varies significantly from project to project 
9 don’t know/decline to answer 
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C5. What is your opinion about the number of regulatory software audits that are involved with bringing a typical 
project to a successful conclusion? Is the number of audits: 

1 too tew to ensure full compliance 3 more than is really needed 

2 about right for ensuring compliance 9 don’t know/decline to answer 

C6. In your experience, what is the typical duration of a regulatory software audit? 

1 1-2 days 4 the duration varies significantly 

2 3-5 days 9 don’t know/decline to answer 

3 mote than one week 

C7. What is your opinion of the typical duration of a regulatory software audit? 

1 too short 4 substantially longer than necessary 

2 about right 9 don’t know/decline to answer 

3 a little longer than necessary 

C9. There are two approaches to getting DO-178B approval: regulatory on-site software audit (location may be 
either at the regulator’s site or the applicant’s site) or desktop review (regulator review of data only, no 
applicant support required). 

How much extra preparation by your team is necessary to support a regulatory on-site software audit over and 
above what a desktop review would require? (Please include the effort of subcontractors and consultants as 
applicable.) 

1 no time 

2 1-2 person-days 

3 3-5 person-days 

4 6-10 person-days 

CIO. Which of the following best describes your opinion about the depth of regulatory software audits you have 
been involved with in the past five years? 

1 Regulatory software audits are superficial and strictly routine 

2 Regulatory software audits are excessively detailed and probing 

3 Regulatory software audits are usually about right 

4 Regulatory software audits vary significantly depending on the reviewer 
9 don’t know/decline to answer 


5 1 1-15 person-days 

6 more than 15 person-days 

9 don’t know/decline to answer 
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SECTION D - YOUR PRIMARY AGO AND OTHER ACOs 

** PROGRAM WILL AUTOMATE THIS SKIP: This section is for airborne respondents only. Ground-based 
respondents, please go to Section GD. 

Dl. Do you interact enough with your ACO to rate their performance regarding software aspects of certification? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” then continue; otherwise skip to Section E 

D2. Overall, how satisfied are you with the job your primary ACO is doing with software aspects of certification? 
Are you . . . 

1 very satisfied 4 somewhat dissatisfied 

2 somewhat satisfied 5 very dissatisfied 

3 neither satisfied nor dissatisfied 9 don’t know/decline to answer 

I-D3. For each of the following attributes, how would you rate your primary ACO? Would you rate each of the 

following as excellent, very good, good, fair, poor, or are you unable to rate the attribute of your primary 
ACO? 

D3a. The first attribute is: returning phone calls 

1 excellent 2 very good 3 good 4 fair 5 poor 9 don’t know 

D3b. The next attribute is: responding to written communication 

1 excellent 2 very good 3 good 4 fair 5 poor 9 don’t know 

D3c. The next attribute is: addressing certification issues 

1 excellent 2 very good 3 good 4 fair 5 poor 9 don’t know 

D3d. The next attribute is: establishing and honoring dates for reviews and approvals 

1 excellent 2 very good 3 good 4 fair 5 poor 9 don't know 

D3e. The next is: coordinating meetings/reviews 

1 excellent 2 very good 3 good 4 fair 5 poor 9 don’t know 

D3g. The next is: coordination with technical resources & national policy making 

1 excellent 2 very good 3 good 4 fair 5 poor 9 don’t know 

D3h. The next is: responding to submitted plans 

1 excellent 2 very good 3 good 4 fair 5 poor 9 don’t know 

D4. Do you have any additional positive or negative comments about your experience working with your primary 
ACO? 

D5. In your opinion, are different companies handled differently within the same ACO? 

1 yes 2 no 9 don’t know/decline to answer 
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D8. Which of the following describes the main reason for using multiple ACOs? 

1 company choice 3 other [please specify:] 

2 customer’s geographic region 9 don’t know/decline to answer 


D9. In the past three years, have you ever received inconsistent software guidance, interpretations, or procedural 
requirements from different ACOs? 

1 yes 2 no 9 don’t know/decline to answer 

If "yes" on D9, continue; otherwise, skip to D20 

D12. How often have you or your team received inconsistent software guidance, interpretations, or procedural 
guidance from two or more ACOs? 

1 once 

2 rarely 

3 occasionally 

D13. How would you rate the general degree of inconsistency between the ACOs you deal with? 

1 they differ only in minor details 

2 they occasionally differ in substantial ways 

3 they seem to have fundamentally different approaches and requirements 
9 don’t know/decline to answer 

D13a. How have the inconsistencies affected your costs? 

1 they’ve caused major additional costs 

2 they’ve caused only minor additional costs 

3 they’ve not added to cost 

4 we’ve saved cost by taking advantage of the alternatives offered by different ACOs 
9 don’t know/decline to answer 


4 frequently 

9 don’t know/decline to answer 
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DIO. Please indicate the area(s) of inconsistent guidance, interpretation, or procedural requirements among ACOs. 
Select all answers that apply. 

1 TSO 

8 

notices and orders 

2 PMA 

9 

verbal direction 

3 TC 

10 

interpretation of DO-178B 

4 STC 

11 

advisory circulars 

5 issue paper 

12 

ACO procedures 

6 special condition 

13 

other [please specify:] 

7 policy memo 

14 

no more/don’t know/decline to answer 

DIOb. And which one of these areas of inconsistency has caused 

you the greatest consequences? 

1 TSO 

8 

notices and orders 

2 PMA 

9 

verbal direction 

3 TC 

10 

interpretation of DO-178B 

4 STC 

11 

advisory circulars 

5 issue paper 

12 

ACO procedures 

6 special condition 

13 

other [please specify:] 

7 policy memo 

14 

don’t know/decline to answer 


D20. In the past three years, have you received inconsistent software guidance from different people within the 
same ACO? 

I yes 2 no 9 don’t know/decline to answer 

If “yes” then continue, otherw ise skip to D25 

D21. From within which ACO have you most often received inconsistent software guidance, interpretations, or 
procedural guidance from different people? 

1 Anchorage Aircraft Certification Office (ACE-l 15N) 

2 Atlanta Aircraft Certification Office (ACE-1 15A) 

3 Boston Aircraft Certification Office (ANE- 150) 

4 Chicago Aircraft Certification Office (ACE-l 15C) 

5 Denver Aircraft Certification Office (ANM-100D) 

6 Engine Certification Office (ANE- 140) 

7 Fort Worth Airplane Certification Office (ASW-150) 

8 Fort Worth Rotorcraft Certification Office (ASW-170) 

9 Fort Worth Special Certification Office ( ASW-190) 

10 Los Angeles Aircraft Certification Office (ANM-100L) 

I I New York Aircraft Certification Office (ANE- 170) 

12 Seattle Aircraft Certification Office (ANM-100S) 

13 Wichita Aircraft Certification Office (ACE-l 15W) 

14 Brussels Aircraft Certification Division (AEU-100) 

16 don’t know/decline to answer 


D22. We’ve found people within this ACO to be inconsistent with one another . . . 


1 once 4 frequently 

2 rarely 9 don’t know/decline to answer 

3 occasionally 
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D23. How have the inconsistencies affected your costs? 

1 they've caused major additional costs 

2 they've caused only minor additional costs 

3 they've not added to cost 

4 we’ve saved cost by taking advantage of the alternatives offered by different people within the ACO 
9 don’t know/decline to answer 

D24. Please indicate the area(s) of inconsistent guidance, interpretation, or procedural requirements you receive 
from different people within the same ACO. Select all answers that apply. 


1 

TSO 

8 

notices and orders 

2 

PMA 

9 

verbal direction 

3 

TC 

10 

interpretation of DO-178B 

4 

STC 

11 

advisory circulars 

5 

issue paper 

12 

ACO procedures 

6 

special condition 

13 

other [please specify:] 

7 

policy memo 

14 

no more/don’t know/decline to answer 


D24a. And which one of these areas of inconsistency has caused you the greatest consequences? 


1 

TSO 

8 

notices and orders 

2 

PMA 

9 

verbal direction 

3 

TC 

10 

interpretation of DO-178B 

4 

STC 

11 

advisory circulars 

5 

issue paper 

12 

ACO procedures 

6 

special condition 

13 

other [please specify:] 

7 

policy memo 

14 

don’t know/decline to answer 


D25. Have you ever been under the impression that you reached a verbal agreement with an ACO about a 
software-related certification issue and later that agreement was nullified by the FAA? 

1 yes 2 no 9 don’t know/decline to answer 

D26. Have you ever had a written agreement with an ACO about a software -related certification issue and later 
that agreement was nullified by the FAA? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” on D25 or D26 continue, otherwise skip to section E 

D27. Which of the following explanations best describes why the agreement changed? Select all answers that 
apply. 

1 the system you developed changed & the certification basis changed 

2 FAA rules changed 

3 FAA personnel changed 

4 company personnel changed 

5 the original FAA parties to the agreement rescinded it 

6 a simple misunderstanding and no one is really to blame 

7 other [please specify:] 

9 no more/don’t know/decline to answer 
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GD4g.The next is: coordination with technical resources & national policy making 

1 excellent 2 very good 3 good 4 fair 5 poor 


GD4h.The next is: responding to submitted plans 

1 excellent 2 very good 3 good 


4 fair 


5 poor 


9 don’t know 


9 don’t know 


GD5. Do you have any additional positive or negative comments about your experience working with your primary 
approving authority? 


GD6. Have you ever used both an AND organization (either AND-740 or AND-730) and AOS-240 for approval of 
software for ground-based systems? 

FAA AND-730: Product Team for the Global Positioning System for the Local Area Augmentation System 
(LAAS) and the Wide Area Augmentation System (WAAS) 

FAA AND-740: Product Team for Navigation and Landing for the Instrument Landing System (ILS) and 
Transponder Landing System (TLS) 

FAA AOS-240: GPS, Navigation and Landing Branch for the Differential Global Positioning Systems for 
Special Category I (SCAT-I) 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” to GD6 continue, otherwise skip to GD21 

GD7. Which of the following best describes the reason for using both approving authorities? 

1 company choice 3 other [please specify:] 

2 customer requirement 9 don’t know/decline to answer 

GD8. Have you ever received inconsistent software guidance, interpretations, or procedural requirements from an 
AND organization (either AND-740 or AND-730) and AOS-240? 

1 yes 2 no 9 don’t know/decline to answer 

GD21.Have you ever been under the impression that you reached a verbal agreement with an approving authority 
about a software-related issue and later that agreement was nullified by the FAA? 

1 yes 2 no 9 don’t know/decline to answer 

GD22.Have you ever had a written agreement with an approving authority about a software-related issue and later 

that agreement was nullified by the FAA? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” on GD21 or GD22 continue, otherwise skip to Section E 

GD23. Which of the following explanations best describes why the agreement changed? Select all answers that 
apply. 

1 the system you developed changed & the certification basis changed 

2 FAA rules changed 

3 FAA personnel changed 

4 company personnel changed 

5 the original FAA parties to the agreement rescinded it 

6 a simple misunderstanding and no one is really to blame 

7 other [please specify:] 

9 no more/don’t know/decline to answer 


70 





SSAC Survey - section e 


SECTION E - QUESTIONS ABOUT INDEPENDENCE 

El. Which one of the following definitions best meets the intent of DO-178B with respect to independence? 

1 the person who verifies the activity is different from the person who produces the artifact under 
consideration 

2 verification must be performed separately from the development organization 
9 don't know/decline to answer 


E2. Overall, how satisfied are you with the requirements for independence in DO-178B? Are you: 


1 very satisfied 

2 somewhat satisfied 

3 neither satisfied nor dissatisfied 


4 somewhat dissatisfied 

5 very dissatisfied 

9 don’t know/decline to answer 


If 2,3,4, or 5 to E2 continue, otherwise skip to E5 


E3. With which independence requirements (if any) do you disagree? If appropriate, it would be helpful to 
mention the Annex A Table, objective, and software level to which your comments relate. 


E5. In your experience, about how big are the software development costs attributed to independence, including 
meeting the objectives in DO-178B? 


1 a negligible cost 

2 a small cost 

3 a substantial cost 


4 a nearly prohibitive cost 

5 unable to estimate 

9 don’t know/decline to answer 


E6. About how much software development time would you estimate is attributed to independence, including 
meeting the objectives in DO-178B? 


1 none or a negligible amount of time 

2 a small amount of time 

3 a substantial amount of time 


4 a nearly prohibitive amount of time 

5 unable to estimate 

9 don’t know/decline to answer 


E7. How much software development time and cost would your company devote to independence if you were not 
required to meet the objectives in DO-178B? 


1 same as now 

2 almost as much as now 

3 substantially less than now 


4 none 

5 unable to estimate 

9 don’t know/decline to answer 


E8. In your opinion, are DO-178B’s requirements for independence: 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 
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SECTION F - MODIFIED CONDITION DECISION COVERAGE 

FO. In the past five years, have you been directly involved in either the development or approval of Level A 
software? 

1 yes 2 no 9 don’t know/decline to answer 

If "yes", continue; otherwise, skip to G1 

I-Fl. MCDC is a level of structural coverage beyond decision coverage, where each condition in a decision must 
be shown to independently affect that decision’s outcome. 

FI . For you and your team, how hard is it to comply with the mcdc objective? 

1 trivially simple 4 extremely difficult 

2 moderately simple 9 don’t know/decline to answer 

3 moderately difficult 

Fib. Which aspects of complying with the mcdc objective are difficult? 

F2. Do you or your team perform mcdc on the source code? 

1 yes 2 no 9 don’t know/decline to answer 

F3. Do you or your team perform mcdc on the object code ? 

1 yes 2 no 9 don’t know/decline to answer 

F4. Which of the following statements best describes your experience with meeting the structural coverage 
objective for mcdc? 

1 We have never found any errors using mcdc. 

2 We have rarely found errors using mcdc that were not found in other testing. 

3 We have occasionally found errors using mcdc that were not found in other testing. 

4 We have frequently found errors using mcdc that were not found in other testing. 

9 don’t know/decline to answer 

If 2,3,4 on F4, continue; otherwise skip to F5 

F4B. Were any of the errors discovered using mcdc safety-related? 

1 yes 2 no 9 don’t know/decline to answer 

F5. For Level A systems, do you perform traceability analysis from source to object code to determine what 
additional verification is necessary for object code that is not directly traceable to the source code? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” to F5 continue, otherwise skip to F8 

F6. Has this resulted in additional verification? 

1 yes 2 no 9 don’t know/decline to answer 

F7. If additional verification did result, were any errors detected? 

1 yes 2 no 9 don’t know/decline to answer 
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If "yes" on F7, continue; otherwise skip to F8 


F7a. Were any of the errors safety-related? 

1 yes 2 no 1 


9 don’t know/decline to answer 


F8. Which of the following best describes the way your team usually achieves structural coverage? 

1 we first do requirements-based testing and use the results to determine if additional structural testing is 
necessary 

2 we develop a test suite to perform structural coverage independent of requirements-based testing 

3 other [please specify:] 

9 don’t know/decline to answer 


F9a. In your team’s experience, about how big are the software development costs attributed to performing mcdc, 
including meeting the objective in DO-178B? 


1 a negligible cost 

2 a small cost 

3 a substantial cost 


4 a nearly prohibitive cost 

5 unable to estimate 

9 don’t know/decline to answer 


F9b. About how much software development time would you estimate is attributed to performing mcdc, including 
meeting the objective in DO-178B? 


1 none or a negligible amount of time 

2 a small amount of time 

3 a substantial amount of time 


4 a nearly prohibitive amount of time 

5 unable to estimate 

9 don’t know/decline to answer 


F9c. How much software development time and cost would your company devote to performing mcdc if you were 
not required to meet the objective in DO- 178B? 


1 same as now 

2 almost as much as now 

3 substantially less than now 

F9d. In your opinion, is DO-178B’s requirement for mcdc: 

1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 none 

5 unable to estimate 

9 don’t know/decline to answer 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 


FI 1 . For Level A systems, how would you rate the overall value of mcdc? 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 
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SECTION G - PRACTICES REGARDING TRACEABILITY 


G1 . Do you or your team use traceability for requirements coverage? 


1 yes 


9 don’t know/decline to answer 


G3. Do you or your team use traceability for regression analysis and testing? 


1 yes 


9 don’t know/decline to answer 


G5. Do you or your team use traceability for change impact analysis? 

1 yes 2 no 9 don’t know/decline to answer 

G7. Do you or your team use traceability only for certification? 


1 yes 


9 don’t know/decline to answer 


G9. Do you or your team use traceability for identifying unnecessary source code ? 


1 yes 


9 don’t know/decline to answer 


G10. In your opinion, is the use of traceability: 

1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 


G 1 1 . Do you or your team document traceability from source code to object code? 
1 yes 2 no 9 don’t know/decline to answer 

If “yes” to Gil then continue, otherwise skip to G13 


G12. Why do you do it? Select all answers that apply. 

1 company procedures 

2 ACO requirement 

3 DER requirement 


4 DO- 178B requirement 

5 other [please specify:] 

9 no more/don’t know/decline to answer 


G13. In your team’s experience, about how big are the software development costs attributed to traceability, 
including meeting the objectives in DO-178B? 


1 a negligible cost 

2 a small cost 

3 a substantial cost 


4 a nearly prohibitive cost 

5 unable to estimate 

9 don’t know/decline to answer 


G14. About how much software development time would you estimate is attributed to traceability, including 
meeting the objectives in DO-178B? 


1 none or a negligible amount of time 

2 a small amount of time 

3 a substantial amount of time 


4 a nearly prohibitive amount of time 

5 unable to estimate 

9 don’t know/decline to answer 


G15. How much software development time and cost would your company devote to traceability if you were not 
required to meet the objectives in DO-178B? 


1 same as now 

2 almost as much as now 

3 substantially less than now 


4 no time as all 

5 unable to estimate 

9 don't know/decline to answer 
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SSAC Survey - section g 


G16. In your opinion, are DO-178B’s requirements for traceability: 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 
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SSAC Survey - section h 


SECTION H - QUALITY ASSURANCE 


I-Hl. Please indicate your level of agreement with these statements: 


How would you rate the overall value of meeting SQA objective 1 (Assurance is obtained that software 
development and integral processes comply with approved software plans and standards) as defined in Table 
A-9 of Annex A in DO-178B? 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 


H2. How would you rate the overall value of meeting SQA objective 2 (Assurance is obtained that transition 
criteria for the software life cycle processes are satisfied) as defined in Table A-9 of Annex A in DO- 178B? 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 


H3. How would you rate the overall value of meeting SQA objective 3 (Software conformity review is conducted) 
as defined in Table A-9 of Annex A in DO-178B? 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 


H4. In your team’s experience, about how big are the software development costs attributed to quality assurance, 
including meeting the objectives in DO-178B? 


1 a negligible cost 

2 a small cost 

3 a substantial cost 


4 a nearly prohibitive cost 

5 unable to estimate 

9 don’t know/decline to answer 


H5. About how much software development time would you estimate is attributed to quality assurance, including 
meeting the objectives in DO-178B? 


1 none or a negligible amount of time 

2 a small amount of time 

3 a substantial amount of time 


4 a nearly prohibitive amount of time 

5 unable to estimate 

9 don’t know/decline to answer 


H6. How much software development time and cost would your company devote to quality assurance if you were 
not required to meet the objectives in DO-178B? 


1 same as now 

2 almost as much as now 

3 substantially less than now 


4 none 

5 unable to estimate 

9 don’t know/decline to answer 


H7. In your opinion, are DO-178B’s requirements for quality assurance: 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 
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17. Please specify the data or documentation you were asked to provide. 


18. How much additional cost has been incurred as a result? 

1 none or a negligible cost 4 a nearly prohibitive cost 

2 a small cost 5 unable to estimate 

3 a substantial cost 9 don’t know /decline to answer 

I8A. Have you ever been asked to provide software data or documentation to an approving authority that was not 
specified in your Plan for Software Aspects of Certification (PSAC) that they approved? 

1 yes 2 no 9 don’t know/decline to answer 
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SSAC Survey - section i 


19. Have you ever been asked to update software data or documentation that had no impact on the approval or 
continued safety or maintenance of the product, at or near the end of the project (for example, a PSAC)? 

1 yes 2 no 9 don’t know/decline to answer 

If "yes” on 19 then continue, otherwise skip to 111 

110. Has approval been delayed as a result? 

1 yes 2 no 9 don’t know/decline to answer 

111. For DO-178B approval of a new software-based system, do you believe that the data requirements to meet 
DO-178B are excessive? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” to 111 then continue, otherwise skip to 113 

112. Which specific data requirements do you believe are unnecessary? 

113. For DO-178B approval of a derivative or follow-on software-based system, do you believe that the data 
requirements to meet DO-178B are excessive? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” to 113 then continue, otherwise skip to 115 

114. Which specific data requirements do you believe are unnecessary? 

115. Is there data required by DO-178B that you provide strictly to meet certification requirements (excluding the 
PSAC, Configuration Index, and Accomplishment Summary) and that you do not use as part of your software 
development process ? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” on 115 then continue, otherwise skip to J1 

116. Please specify which data. 
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J4A. Please specify why you believe tool qualification should not have been required. 

J5. In your team’s experience, about how big are the software development costs attributed to software tool 
qualification, including meeting the objectives in DO-178B? 

1 a negligible cost 4 a nearly prohibitive cost 

2 a small cost 9 don’t know/decline to answer 

3 a substantial cost 

J6. About how much software development time would you estimate is attributed to software tool qualification, 
including meeting the objectives in DO-178B? 

1 none or a negligible amount of time 4 a nearly prohibitive amount of time 

2 a small amount of time 9 don’t know/decline to answer 

3 a substantial amount of time 

J7. How much software development time and cost would your company devote to tool qualification if you were 
not required to meet the objectives in DO-178B? 

1 same as now 4 none 

2 almost as much as now 5 unable to estimate 

3 substantially less than now 9 don't know/decline to answer 
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SSAC Survey - section j 


J8. In your opinion, are DO-178B's requirements for tool qualification . , . 


1 extremely valuable 

2 somewhat valuable 

3 of marginal value 


4 of no value 

5 unable to estimate 

9 don’t know/decline to answer 
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SSAC Survey - section k 


SECTION K - SAFETY 

Kl. Do you believe that DO-178B adequately addresses software -related safety issues? 

1 yes 2 no 9 don’t know/decline to answer 

K2. How often do you or your team perform additional activities outside of those required by DO-178B for 
software-related safety issues? 

1 never 4 for nearly all our projects 

2 rarely - only exceptional projects 5 for every project 

3 for quite a few projects 9 don’t know/decline to answer 

K3. Have you ever worked on a system, developed in compliance with DO-178B, in which a software-related 
system error resulted in a service bulletin or Airworthiness Directive? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” to K3 continue, otherwise skip to K5 

K4. Which of the following best describes the cause(s) of these errors? Select all answers that apply. 

1 Erroneous understanding of high level software requirements 

2 Erroneously developed high level or low level derived requirements 

3 Erroneous software design decisions 

4 Erroneous coding 

5 other [please specify:] 

9 no more/don’t know/decline to answer 

If “no” to K3 continue, otherwise skip to K6 

K5. Which of the following best describes your opinion about the contribution of DO-178B to this absence of 
software-related system error? 

1 DO- 178B made no contribution. 3 DO-178B made a major contribution. 

2 DO- 178B made a minor contribution. 9 don’t know/decline to answer 

K6. Is any safety-related information, other than DO-178B software level, given to your software developers 
regarding the functions they are to implement in software? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” to K6 continue, otherw ise skip to K8 

K7. What is the safety-related information and how is it used? 

K8. Which of the following best describes your team’s procedure for handling derived requirements as defined in 
DO-178B? 

1 to create one or more "place-holder" requirements at a higher level to which the derived requirements can 
be traced 

2 to identify the derived requirements as such without traceability, and add rationale to the documentation 

3 to identify the derived requirements as such, without traceability, and submit them to the safety 
assessment process for review 

4 to find existing and somewhat related requirements at a higher level to which the derived requirements can 
be traced 

5 do not identify the derived requirements as such, and treat them as an artifact of the design process 

6 other [please specify:] 

9 don’t know/decline to answer 
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SSAC Survey - section k 


K9. Have you or your team ever identified any derived requirement that resulted in a safety-related modification 
to the system design? 

1 yes 2 no 9 don’t know/decline to answer 
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SSAC Survey - section l 



L9. Have disagreements arisen among the software DERs that led to schedule delays or wasted resources? 
1 yes 2 no 9 don’t know/decline to answer 


L10. In the past 3 years, have you ever worked on TSO projects where software DERs were used to approve the 
data? 

1 yes 2 no 9 don’t know/decline to answer 
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SSAC Survey - section l 


L 1 1 . In the past 3 years, have you ever had a software DER that disagreed with his predecessor on the project, 
resulting in major project rework? 

1 yes 2 no 9 don’t know/decline to answer 

LI 2. In the past 3 years, has a software DER ever asked you to do anything technical that was not in written FAA 
software policy and guidance? 

[DEFINITION: “Written policy and guidance” refers to the Federal Aviation Regulations, DO-178B, FAA 
notices, FAA Orders, Advisory Circulars, and the FAA policy memos.] 

1 no 3 yes, and the impact was substantial 

2 yes, but the impact was minor 9 don’t know/decline to answer 

L12A. Based on your experience with software DERs, what is your opinion about the degree of delegation to 
software DERs? 

1 there is too little delegation to software DERs 

2 the amount of delegation to software DERs is about right 

3 there is too much delegation to software DERs 

9 don’t know/decline to answer 

LI 3. Have your team’s schedules and costs been adversely affected by the degree of delegation to software DERs? 

1 yes 2 no 9 don’t know/decline to answer 

L15. In the past 3 years, have you been repeatedly dissatisfied with the services provided by a software DER on 
your projects? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes” to LI 5 continue, otherw ise skip to LI 7 

LI 6. Please indicate which ACO appointed the software DER(s) in question? 

1 Anchorage Aircraft Certification Office (ACE-l 15N) 

2 Atlanta Aircraft Certification Office (ACE-1 15 A) 

3 Boston Aircraft Certification Office (ANE-150) 

4 Chicago Aircraft Certification Office (ACE-l 15C) 

5 Denver Aircraft Certification Office (ANM-100D) 

6 Engine Certification Office (ANE-140) 

7 Fort Worth Airplane Certification Office (ASW-150) 

8 Fort Worth Rotorcraft Certification Office (ASW-170) 

9 Fort Worth Special Certification Office (ASW-190) 

10 Los Angeles Aircraft Certification Office (ANM-100L) 

1 1 New York Aircraft Certification Office (ANE-170) 

12 Seattle Aircraft Certification Office (ANM-100S) 

13 Wichita Aircraft Certification Office (ACE-l 15W) 

14 Brussels Aircraft Certification Office (AEU-100) 

15 don’t know/decline to answer 

L17. Has your company ever made a delegation proposal to a software DER that the ACO rejected? 

1 yes 2 no 9 don’t know/decline to answer 

If “yes" to LI 7 continue, otherwise skip to L21 

L18. Why was the delegation proposal rejected? 
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SSAC Survey - section l 


L2 1 . In your opinion, does the FAA provide adequate training for software DERs? 
1 yes 2 no 9 don’t know/decline to answer 


If “no” to L21 continue, otherwise skip to L23 

L22. What additional training should be provided? 

L23. Does your company provide software-DER-specific training? 

1 yes 2 no 9 don’t know/decline to answer 


If “yes” to L23 continue, otherwise skip to Ml 

L24. Which of the following areas are provided? Select all answers that apply. 


1 DER procedures 

2 software engineering 

3 certification process 


4 DO-178B 

5 system engineering 

9 no more/don’t know/decline to answer 
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SSAC Survey - section gl 


SECTION GL - USING DERs FOR GROUND-BASED SYSTEMS 

** PROGRAM WILL AUTOMATE THIS SKIP: If you have been directly involved with software for GROUND- 
BASED systems ("yes" on A5), please answer the questions in this section. If NOT, please go to Section M. 

GL1. In developing a ground-based system using DO-178B for software assurance in the past 3 years, have you 
worked with an individual designated by the FAA as a software DER? 

1 yes 2 no 9 don’t know/decline to answer 

If "yes” to GL1 continue, otherwise skip to Section M 

GL2. Select each of the following ways the software DER was used. Select all answers that apply. 

1 to help review the software architecture and allocate software assurance level 

2 to set up the software development processes to ensure compliance with DO-178B 

3 to review submissions of DO-178B documentation prior to submission to the FAA 

4 other [please specify:] 

9 no more/don’t know/decline to answer 

GL3. Overall, how would you rate your level of satisfaction with your experience with the software DER? Are 
you: 

1 very satisfied 4 somewhat dissatisfied 

2 somewhat satisfied 5 very dissatisfied 

3 neither satisfied nor dissatisfied 9 don’t know/decline to answer 

GL4. Did the use of a software DER as an advisor improve your ability to understand and comply with DO-178B? 

1 yes 2 no 9 don’t know/decline to answer 

GL5. Did the use of a software DER as an advisor reduce the effort involved in making improvements to software 
architecture? 

1 yes 2 no 9 don’t know/decline to answer 

GL6. Did the use of a software DER as an advisor reduce the delay in approval of submissions by the FAA? 

1 yes 2 no 9 don’t know/decline to answer 

GL7. Some have suggested that the FAA should expand the authority of individuals granted software DER 
authority under FAR Part 183, to allow for their use as a DER on ground-based system software assurance. 
This would allow the FAA to rely on the DER to attest to the compliance of a submittal with DO-178B 
objectives. Do you: 

1 strongly agree 4 somewhat disagree 

2 somewhat agree 5 strongly disagree 

3 neither agree nor disagree 9 don’t know/decline to answer 


86 




SSAC Survey - section m 


SECTION M - GETTING INFORMATION YOU NEED 


I-M. You’re near the end of the sun’ey now. 


Ml. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available for obtaining software approvals. 


1 strongly agree 

2 somewhat agree 

3 neither agree nor disagree 


4 somewhat disagree 

5 strongly disagree 

9 don’t know/decline to answer 


MIA. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available for interpreting DO-178B. 


1 strongly agree 

2 somewhat agree 

3 neither agree nor disagree 


4 somewhat disagree 

5 strongly disagree 

9 don’t know/decline to answer 


M2. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available to allow you to establish software levels. 


1 strongly agree 

2 somewhat agree 

3 neither agree nor disagree 


4 somewhat disagree 

5 strongly disagree 

9 don’t know/decline to answer 


M3. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient infonnation is available to assist you with your initial coordination regarding software with the 
approving authority. 


1 strongly agree 

2 somewhat agree 

3 neither agree nor disagree 


4 somewhat disagree 

5 strongly disagree 

9 don’t know/decline to answer 


M4. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available to allow you to prepare for software audits by the approving authority. 


1 strongly agree 

2 somewhat agree 

3 neither agree nor disagree 


4 somewhat disagree 

5 strongly disagree 

9 don’t know/decline to answer 


M5. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available to allow you to obtain final approval from the approving authority. 


1 strongly agree 

2 somewhat agree 

3 neither agree nor disagree 


4 somewhat disagree 

5 strongly disagree 

9 don't know/decline to answer 
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SSAC Survey - section m 


M6. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available to allow you to obtain TSO approvals. 

1 strongly agree 5 strongly disagree 

2 somewhat agree 8 not applicable 

3 neither agree nor disagree 9 don’t know/decline to answer 

4 somewhat disagree 

M7. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available to allow you to obtain PMA approvals. 

1 strongly agree 5 strongly disagree 

2 somewhat agree 8 not applicable 

3 neither agree nor disagree 9 don’t know/decline to answer 

4 somewhat disagree 

M7A. Please indicate whether you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, 
or strongly disagree with the following statement: 

Sufficient information is available on resolving open software issues at final approval (e.g., problem reports). 

1 strongly agree 5 strongly disagree 

2 somewhat agree 8 not applicable 

3 neither agree nor disagree 9 don’t know/decline to answer 

4 somewhat disagree 

M8. When you need information about the software aspects of certification, which of the following sources do 
you personally use? Select all answers that apply. 

1 DO-178B 

2 software DER 

3 DER conferences 

4 web site 

5 public mailings 

6 books in the local library 

M9. By which of the following media do you think information should be made available to you from the FAA? 
Please rank them in order of preference, where l=most preferred medium. 

An entry is required in each field unless a “9” is entered in the first field. Enter ‘9’ in the first field if you can 
not rank these media in terms of your preference 

1 A single web site 

2 Paper 

3 Diskettes 

4 CD-ROM 

M9A. Is there another medium by which you think information should be made available? 

1 yes 2 no 

If “yes” continue, else skip to M10 

M9B. Please specify: 


7 Federal Register 

8 classes run by the FAA 

9 company classes 

10 other [please specify:] 

1 1 no more/don’t know/decline to answer 
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SSAC Survey - section m 


M10. With what style of presentation do you think information should be made available from the FAA? Please 
rank your top three choices in order of preference, where l=most preferred medium. 

Enter rank, then press [ENTER] to move to next item. Enter ‘9’ for any item not in your top three. 

1 documents 

2 interactive video 

3 video tape 

4 web-based training 

5 computer-based training 

6 classroom 

M10A. Is there another style of presentation by which you think information should be made available? 

1 yes 2 no 

If “yes” continue, else skip to next section 

Ml OB. Please specify: 
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SSAC Survey - section p 


SECTION P - APPROPRIATENESS OF DO-178B FOR GROUND-BASED SYSTEMS 

** PROGRAM WILL AUTOMATE THIS SKIP: If you have been directly involved with software for GROUND- 
BASED systems ("yes" on A5), please answer the questions in this section. If NOT, please go to Section Q. 

PI. Do you believe there are qualitative differences in the development of ground-based systems — as contrasted 
with airborne systems — that make DO-178B inappropriate for ground-based systems? 

1 yes, DO-178B would be inappropriate 

2 no, it would be appropriate as is 

3 DO-178B could be adapted to ground-based systems if minor modifications were made 
9 don’t know/decline to answer 

If “yes” to PI continue, otherwise skip to P3 

P2a. For the following area, please indicate if you believe that the assurance mechanisms prescribed by DO-178B 
are appropriate for software used in ground-based systems. 

project planning 

1 yes (DO-178B is appropriate) 2 no 9 don’t know 

P2b. For the following area, please indicate if you believe that the assurance mechanisms prescribed by DO-178B 
are appropriate for software used in ground-based systems. 

requirements definition 

1 yes (DO- 178B is appropriate) 2 no 9 don’t know 

P2c. For the following area, please indicate if you believe that the assurance mechanisms prescribed by DO-178B 
are appropriate for software used in ground-based systems. 

design capture 

1 yes (DO- 178B is appropriate) 2 no 9 don’t know 

P2d. For the following area, please indicate if you believe that the assurance mechanisms prescribed by DO-178B 
are appropriate for software used in ground-based systems. 

coding 

1 yes (DO-178B is appropriate) 2 no 9 don’t know 

P2e. For the following area, please indicate if you believe that the assurance mechanisms prescribed by DO-178B 
are appropriate for software used in ground-based systems. 

reviews, analysis, and testing 

1 yes (DO-178B is appropriate) 2 no 9 don’t know 

P2f. For the following area, please indicate if you believe that the assurance mechanisms prescribed by DO-178B 

are appropriate for software used in ground-based systems. 

coordination of commissioning 

1 yes (DO-178B is appropriate) 2 no 9 don't know 

P3. Do you believe that an independent authority (as the ACO is to an airborne applicant) would be an 
appropriate model for commissioning of ground-based equipment hi the national airspace system? 

1 yes 2 no 9 don't know/decline to answer 


90 




SSAC Survey - section q 


FINAL SECTION - GENERAL COMMENTS 


I-Q. Finally, just a few last questions about your overall views. 

Q3. Suppose you were in charge of the task of developing DO-178C — the next version of DO-178B. What areas 
of this document do you believe are most in need of revision? 


Q4. Thinking of the software approval process in its entirety (not just the written guidelines), what aspects are in 
greatest need of improvement? 

That’s all the questions we have for you. Thank you for taking the time to tell us your experiences and opinions. 
These will be of tremendous help to the technical team in making recommendations to the FAA for streamlining the 
process. 


When you are done with this interview program, please place the floppy disk hi the disk mailer and use the return 
envelope to return it to CSR right away. 

Thanks again! 
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Appendix B 



800 Independence Ave.. S.W 
Washington, DC 20591 

Federal Aviation 
Administration 


U S Deportment 
of Transportation 


MAY I 4 1999 


Dear Members of the Streamlining Software Aspects 
of Certification Team: 


We would like to extend our appreciation to the dedication you have brought to your mission to 
recommend means of streamlining the processes by which software developed for airborne and 
ground-based systems are certified or approved by the Federal Aviation Administration Going 
beyond anecdotal evidence, you have been able to gamer verifiable evidence through your 
workshops and survey. Surveying over seventy companies and receiving a 72% valid return 
rate on the survey is not only a sign of a job well done by the technical team, it demonstrates in 
a measurable way, the level of interest and support that you have gained from the avionics 
community 


The Federal Aviation Administration acknowledges the receipt of your recommendations. 

While these recommendations continue to be reviewed by the FAA, it appears that some of the 
issues are currently being addressed and should be resolved in a short time frame. Additionally, 
your recommendations lend credence to the RTCA Certification Task Force 4 report by 
supporting some of their assertions with hard data with respect to software. At this time we 
would like to extend our commitment to the SSAC, as well as to the participants of the survey 
and workshops, to review each of the recommendations and deliver a response to them by 
September 30, 1999. 


Sincerely, 




Daniel J Mehan 

Assistant Administrator for Information 
Services and Chief Information Officer 



Associate Administrator for Regulation and 
Certification 



Steven J Brown 

Associate Administrator for Air Traffic 
Services 



Associate Administrator for Research and 
Acquisitions 
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